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1 Introduction 

This document is a report about the capabilities and performance of the IOA Toolkit, and in par- 
ticular the tools that provide support for implementing and running distributed systems (checker, 
composer, code generator). The Toolkit compiles distributed systems specified in IOA into Java 
classes, which run on a network of workstations and communicate using the Message Passing Inter- 
face (MPI). In order to test the toolkit, several distributed algorithms were implemented, ranging 
from simple algorithms such as LCR leader election in a ring network to more complex algorithms 
such as the GHS algorithm for computing the minimum spanning tree in an arbitrary graph. All 
of our experiments completed successfully, and several runtime measurements were made. 

2 Experiments 

Implementation Platform The machines used are located in the Theory of Computation Group 
of the MIT Computer Science and Artificial Intelligence Laboratory, forming a Local Area Network. 
They are all Red Hat Linux machines and with Intel Pentium III to IV with clock speed ranging 
from 1 GHz to 3.2 GHz. Even though MPI sets up a connection between every pair of nodes, the 
algorithms only use the communication channels they need. For example, a node (i) in LCR only 
sends to node i+1 and only receives from node i-1. 

2.1 LCR Leader Election 

The algorithm of Le Lann, Chang and Roberts for Leader Election in a ring network was the first 
experiment in running an IOA program on a network of computers. The automaton definition that 
appears in [1] (Section 15.1) was used, with some modifications. For all the algorithms that follow, 
the nodes are automatically numbered from to (size - 1). 

Automata Definitions The automata LCRProcess, LCRNode, SendMediator and ReceiveMe- 
diator were written. The mediator automata are given in Appendix C.l (these automata implement 
the channel automata integrated with MPI functionality). The composed automaton (LCR) ap- 
pears in Appendix C.2 (this is the one that is translated into Java code by the IOA Toolkit). 

LCR Leader Election process automaton 

type Status = enumeration of idle, voting, elected, announced 

automaton LCRProcess (rank : Int , size: Int) 
signature 
input vote 

input RECEIVE(m: Int, const mod(rank - 1, size), const rank: Int) 
output SEND(m: Int, const rank: Int, const mod(rank+l, size)) 



output leader (const rank) 

states 

pending: Mset[Int] := {rank}, 
status: Status := idle 

transitions 
input vote 

eff status : = voting 
input RECEIVE (m, j, i) where m > i 

eff pending := insert (m, pending) 
input RECEIVE (m, j, i) where m < i 
input RECEIVECi , j , i) 

eff status := elected 
output SEND(m, i, j) 

pre status 7^ idle A m 6 pending 

eff pending := deleted, pending) 
output leader (rank) 

pre status = elected 

eff status := announced 



LCR Leader Election composition automaton 



automaton LCRNode (rank : Int , size: Int ) 
components 

P: LCRProcess ( rank , size); 

RM[j:Int]: ReceiveMediat or ( Int , Int, j, rank) 

where j = mod(rank-l, size); 
SM[j:Int]: SendMediator ( Int , Int, rank, j) 

where j = mod(rank+l, size) 



Results The trace of a run on 8 nodes (on 4 machines) can be found in Appendix E.l. A snapshot 
of the trace, representing the last five transitions, is shown below. 



A snapshot of the trace of LCR leader election 



transition: output SEND (7, 5, 6) in automaton LCR (5) 

on condor.csail.mit.edu at 7:25:37:280 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[6 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {7 } Elements removed: {}] }] } 

transition: output RECEIVE (7, 5, 6) in automaton LCR (6) 

on parrot.csail.mit.edu at 7:25:37:755 
Modified state variables: 
P — > Tuple, modified fields: {[pending -> (7)] } 

transition: output SEND (7, 6, 7) in automaton LCR (6) 

on parrot.csail.mit.edu at 7:25:37:770 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 
SM — > Map, modified entries: {[7 -> Tuple, modified fields: {[toSend -> 



Sequence, elements added: {7 } Elements removed: {}] }] } 

transition: output RECEIVE (7, 6, 7) in automaton LCR(7) 

on tui.csail.mit.edu at 7:25:37:872 
Modified state variables: 
P — > Tuple, modified fields: {[status -> elected] } 

transition: output leader (7) in automaton LCR(7) 

on tui.csail.mit.edu at 7:25:37:874 
Modified state variables: 
P — > Tuple, modified fields: {[status -> announced] } 



As the trace indicates, node 7's message has made its way around the ring and eventually 
returned to node 7. At that point, node 7 announced itself as the leader. Figure 2.1 shows the 
messages sent by the nodes. All messages were sent from node i to node i + l(mod 8) and the 
message with value i was sent first. The message with value 7 followed and made the round of the 
ring, to elect node 7 as the leader. 




Figure 2.1: Running the LCR leader election algorithm on a ring of 8 nodes. The white squares 
represent the nodes, while the shaded parallelograms represent the messages sent. 



2.2 Asynchronous Spanning Tree 

The Asynchronous Spanning Tree Algorithm, (see Section 15.3 of [1]) was the next test for the 
Toolkit. The algorithm is still very simple: Given a general graph it computes a spanning tree on 
the graph. This was the first test of the Toolkit on arbitrary graphs, where each node had more 
than one incoming and outgoing communication channels. 



Automata Definitions The AsynchSpanningTree automaton, as defined in [1] (Section 15.3) was 
used. The process automaton and the composition one are listed below. The expanded automaton 
is shown in Appendix C.3. 



Asynchronous Spanning Tree process automaton 



type Message = enumeration of search, null 

automaton sTreeProcess (i : Int) 
signature 

input RECEIVE(m: Message, const i: Int, j: Int) 
output SEND(m: Message, const i: Int, j: Int) 
output PARENKj: Int) 

states 

nbrs : Set [Int] := {}, 

parent: Int := -1, '/, -1 for null 

reported: Bool := false, 

send: Map [Int, Message] 

transitions 

input RECEIVE (m, i, j) 
eff 

if i 7^ A parent = -1 then 
parent := j ; 
for k : Int in nbrs - {j} do 

send [k] := search 
od 
fi 
output SEND(m, i, j) 
pre send[j] = search 
eff send[j] := null 
output PARENKj) 

pre parent = j A reported = false 
eff reported := true 



Asynchronous Spanning Tree composition automaton 



type Message = enumeration of search, null 

automaton sTreeNode (i : Int ) 
components 

P: sTreeProcess ( i ) ; 

RM[j:Int]: ReceiveMediator (Message , Int, i, j) 

SM[j:Int]: SendMediator (Message , Int, i, j) 



Results Figure 2.2 shows a graph of 16 nodes connected in a 4 x 4 grid that was used on some 
of our tests. The source node was node 0. Some of the spanning trees computed are also shown 
in Figure 2.2. A snapshot of the trace follows, showing nodes 1 and 4 sending their message to 
node 5. The message of node 1 arrives first and thus node 5 announces node 1 as its parent. The 
complete trace of this run can be found in Appendix E.2. 

Trace snapshot of the Asynchronous Spanning Tree algorithm 
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Figure 2.2: The 4x4 grid used to test the Asynchronous Spanning Tree Algorithm, along with 2 
different spanning trees computed from 2 different runs of the algorithm. 



transition: output SEND(search 

blackbird . csail .mit . edu 
Modified state variables: 
P — > [nbrs : (0 2 5), parent: 0, 

ioa . runtime . adt . Map Sort] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output SEND(search 

parrot . csail. mit .edu 
Modified state variables: 
P — > [nbrs: (0 5 8), parent: 0. 

ioa . runtime . adt . MapSort] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 



5) in automaton sTreeNode(l) on 



reported: true, send: 



4, 5) in automaton sTreeNode (4) on 



reported: true, send: 



transition: output RECEIVE ( search , 5, 1) in automaton sTreeNode (5) on 

parrot . csail. mit .edu 
Modified state variables: 
P — > [nbrs: (14 6 9), parent: 1, reported: false, send: 

ioa . runtime . adt . MapSort] 
RM — > ioa . runt ime . adt . MapSort 



transition: output PARENT (1) in automaton sTreeNode (5) on 

parrot . csail. mit .edu 
Modified state variables: 

P — > [nbrs: (14 6 9), parent: 1, reported: true, send: 
ioa . runtime . adt . MapSort] 



2.3 Asynchronous Broadcast Convergecast 

This is essentially an extension of the previous algorithm, where along with the construction of a 
spanning tree, a broadcast and convergecast take place (using the computed spanning tree). The 
source node was node 0, and the message 99 (a dummy message) was broadcast on the network. 
Some complexity is added compared to the previous algorithm by the fact that different kinds of 
messages are exchanged. 

Automata Definitions The AsynchBcastAck automaton, as defined in [1] (section 15.3) was 
used. The process automaton and the composition automaton are shown below. The expanded 
automaton is given in Appendix C.4. 

Asynchronous Broadcast Convergecast process automaton 

type Kind = enumeration of beast , ack 

type BCastMsg = tuple of kind : Kind , w : Int 

type Message = union of msg : BCastMsg, kind: Kind 

automaton bcastProcess (rank : Int, nbrs : Set [Int]) 
signature 

input RECEIVE(m: Message, const rank, j: Int) 
output SEND(m: Message, const rank, j: Int) 
internal report (const rank) 

states 

val : Int := -1, '/, -1 = special value denoting null 

parent: Int := -1, 

reported: Bool := false, 

acked: Set [Int] := {>, 

send: Map[Int, Seq [Message ] ] 

initially 

rank = => 

(val = 99 A '/.'/, 99 = the value to be broadcast 

(V j : Int 
((j £ nbrs) => send[j] = O h msg([bcast, val])))) 

transitions 

output SEND(m, rank, j) 
pre m = head ( send [j ] ) 
eff send[j] := tail ( send [j ] ) 
input RECEIVE (m, rank, j) 
eff 

if m = kind(ack) then 

acked := acked U {j} 
else 

if val = -1 then 
val := m . msg . w ; 
parent := j ; 
for k : Int in nbrs - {j} do 

send [k] := send [k] h m 
od 
else 

send[j] := send [ j ] h kind(ack) 



fi 

fi 
internal report (rank) where rank = 
pre acked = nbrs ; 

reported = false 
eff reported := true 
internal report (rank) where rank 7^ 
pre parent 7^ -1; 

acked = nbrs - {parent}; 
reported = false 
eff send [parent] := send [parent] h kind(ack); 
reported := true; 



Asynchronous Broadcast Convergecast composition automaton 

type Message = union of msg : BCastMsg , kind: Kind 

automaton bcastNode (i : Int ) 
components 

P: bcastProcess ( i ) ; 

RM[j:Int]: ReceiveMediat or ( Message , Int, i, j); 

SM[j:Int]: SendMediator (Message , Int, i, j) 



Results The algorithm was tested on several graphs. One of them is shown in Figure 2.3 (a). 
Figure 2.3 (b) and (c) depicts some of the spanning trees that were computed and the communica- 
tion sequence; the numbers next to the nodes represent the sequence at which the nodes reported 
done (through the internal report action). A snapshot of the trace is listed below, showing nodes 
1 and 4 reporting that they are done. At that point, node is enabled and after some communi- 
cation between these nodes, node also reports done. The complete trace of this run is shown in 
Appendix E.3. 

Trace snapshot of the Asynchronous Broadcast Convergecast algorithm 

transition: internal report (1) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (2 5), nbrs: (0 2 5), parent: 0, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 1, 0) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (2 5), nbrs: (0 2 5), parent: 0, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 4, 8) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (5 8), nbrs: (0 5 8), parent: 0, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 

transition: internal report (4) in automaton bcastNode 





























v fo 


• 


v* 


"> 


a 


4 


3 




v fo 


"j 


^ 




\ 







1 




2 




3 









1 




2 




3 









1 




2 




3 










- T* 


t* 


j5 i X 


\* 


\* 


4 




5 




6 




7 




4 




5 




6 




7 




4 




5 




6 




7 










., X* 


T* 


S £ x 


t* 


t* t, 


8 




9 




10 




11 




8 




9 




10 




11 




8 




9 




10 




11 










' X* 


<\ 


\ J 'V J N 


«5 


A " <fe 


1 


2 




1 


3 




14 




15 




12 




13 




14 




15 




12 




13 




14 




15 



Figure 2.3: The grid shown in (a) was used to test the algorithm, (b) and (c) show the spanning 
trees computed in two (different) runs and the numbers next to the nodes indicate the sequence at 
which the nodes reported done. 



Modified state variables: 

P — > [val : 99, acked : (5 8), nbrs : (0 5 8), parent: 0, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 4, 0) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (5 8), nbrs: (0 5 8), parent: 0, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — » ioa . runt ime . adt . MapSort 
SM — » ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 0, 1) in automaton bcastNode 
Modified state variables: 

P — » [val: 99, acked: (1), nbrs: (1 4), parent: -1, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 99]] 
RM — » ioa . runt ime . adt . MapSort 
SM — » ioa . runtime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 0, 4) in automaton bcastNode 
Modified state variables: 

P — » [val: 99, acked: (1 4), nbrs: (1 4), parent: -1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 99]] 
RM — » ioa . runt ime . adt . MapSort 



transition: internal report (0) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (1 4), nbrs: (1 4), parent: -1, reported: true 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 99]] 



2.4 Leader Election Using Broadcast Convergecast 

In page 500 of [1], the author describes how the Asynchronous Broadcast Convergecast algorithm 
can be used to implement a leader election algorithm on a general graph using the Asynchronous 
Broadcast Convergecast algortithm. The main idea is to have every node act as a source node 
and create its own spanning tree, broadcast its UID using this spanning tree and hear from all the 
other nodes via a convergecast. During this convergecast, along with the acknowledge message, the 
children also send what they consider as the maximum UID in the network. The parents gather 
the maximum UIDs from the children, compare it to their own UID and send the maximum to 
their own parents. Thus, each source node learns the maximum UID in the network and the node 
whose UID equals the maximum one announces itself as a leader. 

Automata Definitions The process and composition automata are shown below. The expanded 
automaton is defined in Appendix C.5. 

Leader Election Using Broadcast Convergecast process automaton 

type Kind = enumeration of beast , ack 

type BCastMsg = tuple of kind: Kind, w: Int 

type AckMsg = tuple of kind : Kind , mx : Int 

type MSG = union of bmsg : BCastMsg, amsg : AckMsg, kind: Kind 

type Message = tuple of msg : MSG, source: Int 

automaton bcastLeaderProcess (rank : Int) 
signature 

input RECEIVE(m: Message, i: Int, j: Int) 
output SEND(m: Message, i: Int, j: Int) 
internal report (i: Int, source: Int) 
internal finished 
output LEADER 
states 

nbrs : Set [Int ] , 

val : Map [Int, Int], '/, initialially -1 (null) for all nodes 

parent: Map [Int, Int], '/, initialially -1 (null) for all nodes 

reported: Map [Int, Bool], '/, initialially false for all nodes 

acked : Map [Int, Set [Int]], '/, initialially {} for all nodes 

send: Map [Int, Int, Seq [Message ]] , '/, First variable: source. 

max: Map [Int, Int], '/, The max value found in the network , initially i 
elected: Bool := false, 
announced: Bool := false 
initially 
(V j : Int 

((0 < j A j < 16) => 
(rank = j => val[j] = rank 
A rank / j => val[j] = -1 
A parent [ j ] = -1 
A acked[j] = {} 
A max [ j ] = rank 
A(V k: Int 

(send [j ,k] = {} A 
(k 6 nbrs A rank = j ) =► 
send[j, k] = {} h [bmsg([bcast , 99]), j])))) 
transitions 

10 



output SEND(m, i, j) 

pre m = head ( send [m . source , j]) 

eff send [m . sour ce , j] := tail ( send [m . sour ce , j]) 
input RECEIVE (m, i, j) 

eff if m.msg = kind(ack) then 

acked [m . source ] := acked [m . sour ce ] U {j} 
elseif tag(m.msg) = amsg then 

if max [m . sour ce ] < m . msg . amsg . mx then 

max [m . sour ce ] := m . msg . amsg . mx ; 
f i ; 

acked [m . source ] := acked [m . source] U {j} 
else '/.BcastMsg 

if val [m . source] = -1 then 

val [m . source ] := m . msg . bmsg . w ; 

par ent [m . sour ce ] := j; 

for k:Int in nbrs - {j} do 

send [m . sour ce , k] := send [m . source , k] h m 
od 
else 

send [m . sour ce , j] := send [m . source , j] h [kind(ack), m. source] 
fi 
fi 
internal finished 

pre acked [rank] = nbrs A 

reported [rank] = false 
eff reported [rank] := true; 

if (max [rank] = rank) then 
elected := true 

f i ; 

output LEADER 

pre elected = true A announced = false 
eff announced := true 
internal report (i, source) where i 7^ source 
pre parent [source] 7^ -1 A 

acked [ source ] = nbrs - {parent [source]} A 
r eport ed [ sour ce ] = false 
eff send[source, parent [ sour ce ] ] : = 

send[source, parent [source ] ] h [amsg([ack, max [ sour ce ]]) , source]; 
reported [ sour ce ] := true; 



Leader Election Using Broadcast Convergecast composition automaton 

automaton bcastLeaderNode (i : Int ) 
components 

P: bcastLeaderProcess ( i ) ; 

RM[j:Int]: ReceiveMediat or ( Message , Int, i, j); 

SM[j:Int]: SendMediator (Message , Int, i, j) 



Results This algorithm was also tested on the 4x4 grid that was used in the two previous 
algotithms (see Figure 2.3 (a)). The algorithm terminated correctly and announced node 15 as the 
leader. A snapshot of the trace is shown below. The complete trace of this run can be found on 
the web, at http : //theory . csail.mit .edu:/~pmavrom/ioaReport .html 

11 



Trace snapshot of the Leader Election Using Broadcast Convergecast algorithm 

transition: output RECEIVE ( [msg : amsg ( [kind : ack , mx : 13]), source: 14], 

14, 10) in automaton bcastLeader ( 14) on drake.csail.mit.edu at 9:07:20:455 
Modified state variables: 

P — > Tuple, modified fields: {[acked -> Map, modified entries: {[14 -> (10 

13 15)]}] } 
SM — > Map, modified entries: {[10 -> Tuple, modified fields: {[toSend -> 

Sequence, elements added: {} Elements removed: {[msg: amsg ([kind: ack, mx : 14]) 

source: 11] }] [sent -> Sequence, elements added: {[msg: amsg([kind: ack, mx : 

14]), source: 11] } Elements removed: {}] }] } 
c2 -> 7 

transition: internal finished () in automaton bcastLeader (14) 

on drake.csail.mit.edu at 9:07:20:464 
Modified state variables: 
P — > Tuple, modified fields: {[reported -> Map, modified entries: {[14 -> 

true]}] } 
c2 -> 14 
k -> 15 

transition: output RECEIVE ( [msg : amsg ([kind: ack, mx : 14]), source: 15], 

15, 11) in automaton bcastLeader (15) on loon.csail.mit.edu at 9:07:20:684 
Modified state variables: 

P — > Tuple, modified fields: {[acked -> Map, modified entries: {[15 -> (11 

14)]}] } 
SM — > Map, modified entries: {[11 -> Tuple, modified fields: {[toSend -> 

Sequence, elements added: {} Elements removed: {[msg: amsg ([kind: ack, mx : 15]) 

source: 4] }] [sent -> Sequence, elements added: {[msg: amsg([kind: ack, mx : 

15]), source: 4] } Elements removed: {}] }] } 
c2 -> 7 

transition: internal finished () in automaton bcastLeader (15) 

on loon.csail.mit.edu at 9:07:20:687 
Modified state variables: 
P — > Tuple, modified fields: {[reported -> Map, modified entries: {[15 -> 

true]}] [elected -> true] } 
c2 -> 15 

transition: output LEADER () in automaton bcastLeader (15) 

on loon.csail.mit.edu at 9:07:20:689 
Modified state variables: 
P — > Tuple, modified fields: {[announced -> true] } 



2.5 Unrooted Spanning Tree to Leader Election 

The algorithm STtoLeader of [l](page 501) was implemented as the next test for the Toolkit. The 
algorithm takes as input an unrooted spanning tree and returns a leader. The automaton listed 
below was written, according to the description of the algorithm. 

Automata Definitions The process and composition automata are listed below, while the ex- 
panded automaton can be found in Appendix C.6. 
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Unrooted Spanning Tree to Leader Election process automaton 

type Status = enumeration of idle, elected, announced 
type Message = enumeration of elect 
axioms ChoiceSet ( Int for E) 

automaton sTreeLeaderProcess (rank : Int, nbrs : Set [Int ] ) 
signature 

input RECEIVE(m: Message, const rank: Int, j: Int) 
output SEND(m: Message, const rank: Int, j: Int) 
output leader 
states 

receivedElect : Set [Int] , 
sentElect : Set [Int] , 
status : Status , 
send: Map[Int, Seq [Message ] ] 
initially 
true => 

receivedElect = {} 
A sentElect = {}■ 
A status = idle 
A (size(nbrs) = 1 => 

send [chooseRandom (nbrs ) ] 
= send [chooseRandom (nbrs ) ] h elect) 
transitions 

input RECEIVE (m, i, j) 

eff receivedElect := insert(j, receivedElect); 

if size ( receivedElect ) = size(nbrs) - 1 then 
send [ chooseRandom (nbrs - receivedElect)] : = 

send [ chooseRandom (nbrs - receivedElect)] h elect; 
sentElect := 

insert ( chooseRandom (nbrs - receivedElect), sentElect) 
elseif receivedElect = nbrs then 

if j £ sentElect then if i > j then status := elected fi 
else status := elected 
fi 
fi 
output SEND(m, i, j) 
pre m = head ( send [j ] ) 
eff send[j] := tail ( send [j ] ) 
output leader 

pre status = elected 
eff status := announced 



Unrooted Spanning Tree to Leader Election composition automaton 

automaton sTreeLeaderNode (rank : Int , nbrs : Set [Int ] ) 
components 

P: sTreeLeaderProcess (rank , nbrs); 

RM[j:Int]: ReceiveMediat or ( Message , Int, rank, j); 

SM[j:Int]: SendMediator (Message , Int, rank, j) 



Results The algorithm was tested on several graphs and different spanning trees as input. A 
spanning tree on a 4 x 4 grid is shown in Figure 2.4. The algorithm worked correctly, announcing 
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only one node as the leader. A snapshot of the trace is listed below. In that specific run, the edge 
between nodes 5 and 6 had elect messages sent in both directions, with node 6 being elected as the 
leader, having a larger UID. The complete trace of this run can be found in Appendix E.4. 
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Figure 2.4: (a) shows the unrooted spanning tree used in one run. In (b), the arrows show the 
direction of the messages in a specific run, where the edge between nodes 5 and 9 had elect messages 
sent in both directions. Node 9, with a larger UID was elected as the leader. 



Trace Snapshot of the Unrooted Spanning Tree to Leader Election 



transition: output SEND(elect, 6, 5) in automaton sTreeLeader (6) 

on drake.csail.mit.edu at 9:29:15:112 

Modified state variables: 

P — > Tuple, modified fields: {[send -> Map, modified entries: {[5 -> 



Sequence, elements added: 
SM — > Map, modified entries: 

Sequence, elements added: 
k -> 5 
tempNbrs2 — > (10 2) 



{} Elements removed: -[elect }] }] } 

{[5 -> Tuple, modified fields: -[[toSend 

{elect } Elements removed: {}] }■] )• 



transition: output SEND(elect, 5, 6) in automaton sTreeLeader (5) 

on loon.csail.mit.edu at 9:29:16:182 

Modified state variables: 

P — > Tuple, modified fields: {[send -> Map, modified entries: {[6 -> 

Sequence, elements added: {} Elements removed: {elect }] }] "} 
SM — > Map, modified entries: {[6 -> Tuple, modified fields: {[toSend -> 

Sequence, elements added: {elect } Elements removed: {}] }] } 
k -> 6 
tempNbrs2 — > (14 9) 

transition: output RECEIVE ( elect , 5, 6) in automaton sTreeLeader (5) 

on loon.csail.mit.edu at 9:29:16:494 

Modified state variables: 

P — > Tuple, modified fields: { [receivedElect -> (1 4 6 9)] } 

SM — > Map, modified entries: {[6 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {} Elements removed: {elect }] 
[sent -> Sequence, elements added: {elect } Elements removed: {}] }] } 
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transition: output RECEIVE ( elect , 6, 5) in automaton sTreeLeader (6) 
on drake.csail.mit.edu at 9:29:16:554 

Modified state variables: 

P — > Tuple, modified fields: { [receivedElect -> (10 2 5)] [status -> 
elected] } 

SM — > Map, modified entries: {[5 -> Tuple, modified fields: {[toSend 
Sequence, elements added: {} Elements removed: -[elect }] [sent -> 
Sequence, elements added: {elect } Elements removed: {}] }] } 

transition: output leader () in automaton sTreeLeader (6) 

on drake.csail.mit.edu at 9:29:16:559 

Modified state variables: 

P — > Tuple, modified fields: {[status -> announced] } 



2.6 GHS Minimum Spanning Tree 

The last algorithm we ran using the Toolkit was the algorithm of Gallager, Humblet and Spira for 
finding the minimum-weight spanning tree in an arbitrary graph. Welch, Lamport and Lynch give 
an I/O automaton definition of the GHS algorithm in [2] , which is used as a basis for our automata 
definitions. One technical detail in running this version of the algorithm is that the edge weights 
needed to be unique. This algorithm is significantly longer than the previous ones, with 7 different 
messages exchanged and 12 state variables for each automaton. 

Automata Definitions The automaton below defines the algorithm and the composition with 
the mediator automata is shown after that. The expanded automaton can be found in Ap- 
pendix C.7. 

GHS process automaton 

type Nstatus = enumeration of sleeping, find, found 

type Edge = tuple of s : Int , t : Int 

type Link = tuple of s : Int , t : Int 

type Lstatus = enumeration of unknown, branch, rejected 

7, Message Types 

type Msg = enumeration of CONNECT, INITIATE, TEST, REPORT, ACCEPT, 

REJECT, CHANGER00T 
type ConnMsg = tuple of msg : Msg , 1 : Int 
type Status = enumeration of find, found 

type InitMsg = tuple of msg: Msg, 1: Int, c: Null [Edge], st : Status 
type TestMsg = tuple of msg: Msg, 1: Int, c: Null [Edge] 
type ReportMsg = tuple of msg : Msg , w : Int 
type Message = union of connMsg : ConnMsg, initMsg : InitMsg, 

testMsg : TestMsg , reportMsg : ReportMsg , 

msg : Msg 
uses ChoiceSet (Link ) 

7.7. 

7, automaton GHSProcess : Process of GHS Algorithm for min . spanning tree 

'/, rank: The UID of the automaton, automatically initialized by MPI 

'/, size: The size of the network, automatically initialized by MPI 

'/, links: Set of Links with source = rank (L_p(G)) 
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% weight: Maps the Links G links to their weight 
VI. 

automaton GHSProcess (rank : Int , size: Int , links: Set [Link], weight: Map [Link, Int]) 
signature 

input startP 

input RECEIVE(m: Message, const rank, i: Int) 

output InTree (1 : Link) 

output NotInTree(l: Link) 

output SEND(m: Message, const rank, j: Int) 

internal ReceiveConnect (qp : Link, l:Int) 

internal Receivelnit iate (qp : Link, 1 : Int , c: Null [Edge], st : Status) 

internal ReceiveTest (qp : Link, 1 : Int , c: Null [Edge]) 



nternal ReceiveAccept (qp 
nternal ReceiveRe j ect (qp 
nternal ReceiveReport (qp 



Link) 
Link) 
Link , w : 



Int) 



internal ReceiveChangeRoot (qp : Link) 



states 

nstatus: Nstatus, 
nf rag : Null [Edge], 
nlevel : Int , 
bestlink: Null [Link] , 
bestwt : Int , 
testlink: Null [Link] , 
inbranch : Link , 
findcount : Int, 
lstatus: Map [Link , Lstatus], 
queueOut : Map [Link , Seq [Message ]] , 
queueln : Map [Link , Seq [Message ]] , 
answered: Map [Link , Bool], 
'/, Temporary variables 

min : Int, minL : Null [Link], S: Set [Link] 
initially 

true => nstatus = sleeping 
A nfrag = nil 
A nlevel = 

A bestlink = embed ( chooseRandom ( links ) ) 
A bestwt = weight [ chooseRandom ( links ) ] 
A testlink = nil 

A inbranch = chooseRandom ( links ) 
A findcount = 
A V 1 : Link 

(1 G links => 

lstatus [1] = unknown 
A answered [1] = false 
A queueOut [1] = {} 
A queueln [1] = {}) 
transitions 
7. INPUT ACTIONS 
input startP 

eff if nstatus = sleeping then 
'/.Wake Up 

minL := embed ( chooseRandom ( links )) ; min 
for tempL:Link in links do 

if weight [tempL] < min then minL := embed(tempL) 
ocl ; 



= weight [minL . val] ; 



weight [tempL] fi 
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lstatus [minL . val] := branch; nstatus := found; 

queueOut [minL . val] := queueOut [minL . val] h connMsg ( [CONNECT , 0]); 
•/.EndWakeUp 
fi 
input RECEIVE (m: Message, i : Int , j:Int) 

eff queueln [ [i , j ] ] := queueln [ [i , j ] ] h m 
7, OUTPUT ACTIONS 
output InTree(l: Link) 

pre answered [1] = false A lstatus [1] — branch 
eff answered [1] := true 
output NotInTree(l: Link) 

pre answered [1] = false A lstatus [1] = rejected 
eff answered [1] := true 
output SEND(m: Message, i: Int, j: Int) 
pre m = head (queueOut [ [i , j ]] ) 

eff queueOut [ [i , j ] ] := tail ( queueOut [[ i , j ]] ) 
•/, INTERNAL ACTIONS 
internal ReceiveConnect (qp : Link, 1: Int) 

pre head(queueln [qp] ) = connMsg ([ CONNECT , 1]) 
eff queueln [qp] := tail ( queueln [qp] ) ; 
if nstatus = sleeping then 
'/.Wake Up 

minL := embed ( chooseRandom ( links )) ; min := weight [minL . val] ; 
for tempL:Link in links do 

if weight [tempL] < min then minL := embed ( tempL ) ; min := weight [tempL] fi 
od ; 

1st atus [minL . val] := branch; nstatus := found; 

queueOut [minL . val] := queueOut [minL . val] h connMsg ( [CONNECT , 0]); 
7, EndWakeUp 

f i ; 

if 1 < nlevel then 

lstatus [[qp.t.qp.s]] := branch; 
if testlink 7^ nil then 

queueOut [ [qp . t , qp . s] ] := queueOut [ [qp . t , qp . s] ] h initMsg ([ INITIATE , 

nlevel, nfrag, find]); 
findcount := findcount + 1 
else 

queueOut [ [qp . t , qp . s] ] := queueOut [ [qp . t , qp . s] ] h initMsg ([ INITIATE , 
nlevel, nfrag, found]) 

f i ; 
else 

if 1st atus [ [qp . t , qp . s] ] = unknown then 

queueln [qp] := queueln [qp] h connMsg ( [CONNECT , 1]) 
else 

queueOut [ [qp . t , qp . s ] ] := queueOut [ [qp . t , qp . s] ] h initMsg ([ INITIATE , 
nlevel+1, embed([qp.t, qp.s]), find]) 
fi 
fi 
internal Receivelnit iate (qp : Link, 1 : Int , c: Null [Edge], st : Status) 
pre head (queueln [qp] ) = initMsg ([ INITIATE , 1, c, st]) 
eff queueln [qp] := tail ( queueln [qp] ) ; 
nlevel := 1 ; 
nfrag := c ; 

if st = find then nstatus := find else nstatus := found fi ; 
'/, - Let S = { [p,q] : 1st atus [ [p , r] ] = branch, r 7^ q} - 
S := {>; 
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for pr : Link in links do 

if pr . t 7^ qp . s A lstatus [pr] = branch then 

S := S U {pr} 
fi 
od; 
for k : Link in S do 

queueOut [k] := queueOut [k] h initMsg ([ INITIATE , 1, c, st]) 
od; 
if st = find then 

inb ranch := [qp.t, qp.s]; 

bestlink := nil; 

bestwt := 10000000; '/. Infinity 

•/.TestP 

minL := nil; min := 10000000; '/. Infinity 

for tempL:Link in links do 

if weight [tempL] < min A 1st atus [ t empL] = unknown then 

minL := embed ( tempL ) ; min := weight [tempL] 
f i ; 
od; 

if minL 7^ nil then 
testlink := minL; 

queueOut [minL . val] := queueOut [minL . val] h testMsg ( [TEST , nlevel , nf rag] ) ; 
else 

testlink := nil ; 
/.Report 

if findcount = A testlink = nil then 
nstatus := found; 

queueOut [ inbranch] := queueOut [ inbranch] h reportMsg ( [REPORT , bestwt]) 
fi 

'/.EndReport 
f i ; 

'/.EndTestP 

findcount := size(S) 
fi 

internal ReceiveTest (qp : Link, 1: Int , c: Null [Edge]) 
pre head(queueln [qp] ) = testMsg ( [TEST , 1, c]) 
e f f queue In [qp] := tail ( queue In [qp] ) ; 
if nstatus = sleeping then 
7. Wake Up 

minL := embed ( chooseRandom ( links )) ; min := weight [minL . val] ; 
for tempL:Link in links do 

if weight [tempL] < min then minL := embed ( tempL ) ; min := weight [tempL] f i ; 
od; 

1st atus [minL . val] := branch; nstatus := found; 

queueOut [minL . val] := queueOut [minL . val] h connMsg ( [CONNECT , 0]); 
•/.EndWakeUp 

f i ; 

if 1 > nlevel then 

queueln[qp] := queueln[qp] h testMsg ( [TEST , 1, c]); 
else 

if c / nfrag then 

queueOut [ [qp . t , qp.s]] := queueOut [ [qp . t , qp.s]] h msg(ACCEPT) 
else 

if lstatus [ [qp . t , qp.s]] = unknown then lstatus [ [qp . t , qp.s]] := rejected fi 
if testlink 7^ embed([qp.t, qp.s]) then 
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queueOut [ [qp . t , qp . s ] ] := queueOut [ [qp . t , qp.s]] h msg(REJECT) 
else 
•/.Test 

minL := nil; min := 10000000; */. Infinity 
for tempL:Link in links do 
if weight [tempL] < min A 1st atus [ t empL] = unknown then 

minL := embed(tempL); min := weight [tempL] 
f i ; 
od; 

if minL 7^ nil then 
testlink := minL; 

queueOut [minL . val] := queueOut [minL . val ] h testMsg ( [TEST , nlevel , nf rag] ) 
else 

testlink := nil; 
"/Report 

if findcount = A testlink = nil then 
nstatus := found; 

queueOut [ inbranch] := queueOut [ inbranch] h reportMsg ( [REPORT , bestwt]) 
fi 
•/EndReport 

f i ; 

•/EndTest 
f i ; 



f i ; 
f i ; 

internal ReceiveAccept (qp : Link) 

pre head(queueln [qp] ) = msg(ACCEPT) 
eff queueln[qp] := tail ( queue In [qp] ) ; 
testlink := nil ; 

if weight [ [qp . t , qp.s]] < bestwt then 
bestlink := embed([qp.t, qp.s]); 
bestwt := weight [ [qp . t , qp.s]]; 

f i ; 

"/Report 

if findcount = A testlink = nil then 

nstatus := found; 

queueOut [ inbranch ] := queueOut [ inbranch] h reportMsg ( [REPORT , bestwt]) 
fi 
"/EndReport 

internal ReceiveRe j ect (qp : Link) 

pre head (queueln [qp] ) = msg(REJECT) 
eff queueln [qp] := tail ( queueln [qp] ) ; 

if 1st atus [ [qp . t , qp.s]] = unknown then 1st atus [ [qp . t , qp.s]] := rejected f i ; 

"/Test 

minL := nil; min := 10000000; '/, Infinity 

for tempL:Link in links do 

if weight [tempL] < min A lstatus [tempL] = unknown then 
minL := embed ( tempL ) ; min := weight [tempL] 

f i ; 
od; 

if minL 7^ nil then 

testlink := minL; 

queueOut [minL . val ] := queueOut [minL . val] h testMsg ( [TEST , nlevel, nfrag]); 
else 
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testlink := nil; 
'/, R e p o r t 
if findcount = A testlink = nil then 
nstatus := found; 

queueOut [inbranch] := queueOut [inbranch] h reportMsg ( [REPORT , bestwt]) 
fi 
'/.EndReport 
fi 
'/.EndTest 

internal ReceiveReport (qp : Link, w: Int) 

pre head (queueln [qp] ) = reportMsg ( [REPORT , w] ) 
eff queueln[qp] : = tail ( queueln [qp] ) ; 
if [qp.t, qp.s] 7^ inbranch then 
findcount := findcount -1; 
if w < bestwt then 
bestwt := w; 
bestlink := embed ( [qp . t , qp.s]) 

f i ; 

'/, R e p o r t 

if findcount = A testlink = nil then 
nstatus := found; 

queueOut [ inbranch] := queueOut [ inbranch] h reportMsg ( [REPORT , bestwt]) 
fi 

'/.EndReport 
else 

if nstatus = find then 

queueln [qp] := queueln [qp] h reportMsg ( [REPORT , w] ) 
elseif w > bestwt then 
'/.ChangeRoot 
if 1st atus [best link . val] = branch then 

queueOut [bestlink . val] := queueOut [best link . val ] h msg ( CHANGEROOT ) 
else 

queueOut [best link . val ] := queueOut [best link . val] h 

connMsg ( [CONNECT , nlevel]) ; 
1st atus [bestlink . val] := branch 
fi 

'/EndChangeRoot 
fi 
fi 

internal ReceiveChangeRoot (qp : Link) 

pre head(queueln [qp] ) = msg ( CHANGEROOT ) 
eff queueln [qp] := tail ( queueln [qp] ) ; 
'/.ChangeRoot 
if lstatus [bestlink . val] = branch then 

queueOut [bestlink . val] := queueOut [bestlink . val] h msg ( CHANGEROOT ) 
else 

queueOut [bestlink . val ] := queueOut [bestlink . val] h connMsg ( [CONNECT , nlevel]) 
1st atus [best link . val] := branch 
fi 
"/EndChangeRoot 

GHS composition automaton 
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automaton GHSNode (rank : Int , size: Int , links: Set [Link], weight: Map [Link , Int]) 
components 

P: GHSProcess ( rank , size, links, weight); 
RM[j:Int]: ReceiveMediat or ( Message , Int, rank, j); 
SM[j:Int]: SendMediator (Message , Int, rank, j) 



Results One of the graphs that were used to test the algorithm is shown in Figure 2.5 (a). The 
(unique) minimum spanning tree computed by the algorithm is shown in Figure 2.5 (b). A snap- 
shot of the trace follows, showing the beginning of the algorithm with node waking up (after a 
start action), sending a CONNECT message to its minimun weight outgoing edge (1) and reporting 
this edge as part of the minimum spanning tree. The complete trace of this run can be found 
at http : //theory . csail.mit.edu :/~pmavrom/ioaReport .html. The algorithm ran correctly, re- 
turning the unique minimum spanning tree in all cases. 
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Figure 2.5: (a) shows one of the graphs used to test the GHS algorithm. In (b), the (unique) 
minimum spanning tree computed by the algorithm 

Trace snapshot of the GHS algorithm 

Initialization starts (0) on loon.csail.mit.edu at 7:05:55:830 

Modified state variables: 

P — > [nstatus: sleeping, nf rag : nil, nlevel : 87, bestlink : nil, bestwt : 
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87, testlink: nil, inbranch : [s: 87, t: 87], findcount: 87, lstatus : Map{}, 
queueOut : Map{}, queueln: Map{}, answered: Map{}, min : 87, minL : nil, S: ()] 
RM -> Map{> 

0, t : 4]) 



SM -> 


Map{> 




links 


-> ([s: 0, 


t: 1] [s 


Ink - 


> [s: 87, t 


: 87] 


Inks 


-> () 




rank 


-> null 




size 


-> null 




t empL 


-> [s: 87, 


t : 87] 


t empL: 


Inks — » () 





weight -> Map{[[s: 0, t: 1] -> 8] [[s: 0, t: 4] -> 12] } 

Initialization ends 

transition: input startP () in automaton GHS (0) 
on loon.csail.mit.edu at 7:05:55:852 

Modified state variables: 

P — > [nstatus: found, nfrag: nil, nlevel : 0, bestlink: embed([s: 0, t: 

4]), bestwt : 12, testlink: nil, inbranch: [s: 0, t: 1], findcount: 0, lstatus: 
Map{[[s: 0, t: 1] -> branch] [[s: 0, t: 4] -> unknown] }, queueOut: Map{[[s: 0, 
t: 1] -> {connMsgC [msg : CONNECT, 1: 0])}] [[s: 0, t: 4] -> {}] >, queueln: 
Map{[[s: 1, t: 0] -> {}] [[s: 4, t: 0] -> {}] }, answered: Map{[[s: 0, t: 1] -5 
false] [[s: 0, t: 4] -> false] }, min: 8, minL: embed([s: 0, t: 1]), S: ()] 
RM — > Map{[l -> [status: idle, toRecv: {}, ready: false]] [4 -> [status: 
idle, toRecv: {}, ready: false]] } 

SM — > Map{[l -> [status: idle, toSend: {} , sent: {}, handles: {}] ] [4 -> 
[status: idle, toSend: {}, sent: {}, handles: {}]] } 

0, t : 4]) 



links - 


-> ([s: 


0, t: 1] [s: 


Ink -> 


[s: 87 


', t: 87] 


Inks — > 


() 




rank — » 







size — > 


16 




tempL - 


-* [s: 


0, t: 4] 


t empLin 


iks — > 


() 


weight 


— > Map 


,{[[s: 0, t : 1] 



-> 8] [[s : 0, t : 4] -> 12] } 

transition: output SEND ( connMsg ( [msg : CONNECT, 1: 0]), 0, 1) in automaton 

GHS(0) on loon.csail.mit.edu at 7:05:55:864 
Modified state variables: 
P — > Tuple, modified fields: {[queueOut -> Map, modified entries: {[[s: 0, 

t: 1] -> Sequence, elements added: {} Elements removed: { connMsg ( [msg : CONNECT, 

1: 0]) }]}] } 
SM — > Map, modified entries: {[1 -> Tuple, modified fields: {[toSend -> 

Sequence, elements added: { connMsg ( [msg : CONNECT, 1: 0]) } Elements removed: {}] 

}]} 
Ink — > Tuple, modified fields: {[s -> 0] [t -> 1] > 

transition: output InTree([s: 0, t: 1]) in automaton GHS (0) 

on loon.csail.mit.edu at 7:05:55:915 
Modified state variables: 
P — > Tuple, modified fields: {[answered -> Map, modified entries: {[[s: 0, 

t : 1] -> true] }] > 
SM — > Map, modified entries: {[1 -> Tuple, modified fields: {[toSend -> 

Sequence, elements added: {} Elements removed: { connMsg ( [msg : CONNECT, 1: 0]) }] 

[sent -> Sequence, elements added: { connMsg ( [msg : CONNECT, 1: 0]) } Elements 

removed : {}■] }] } 
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3 Results 

Figures 3.6, 3.7 and 3.8 depict the runtime results of the algorithms on 1 node per machine, a 
constant number of nodes and a constant number of machines respectively. The runtimes were 
measured in seconds, from the time when the first node started initialization until the time at 
which the algorithm terminated, and they are averages on 10 runs. The tables in Appendix D show 
the raw data collected from our experiments. In Figure 3.6(a) to (b) the relationship between time 
and number of nodes is not very clear due to the short time taken, number of nodes and messages 
exchanged. Figure 3.6(d) however, where the number of messages exchanged is large shows the 
clear (linear) relation. 

4 Conclusions 

During the months of June and July 2004 we had the opportunity to test the capabilities and the 
performance of the version of the toolkit that was then available. We started by mostly modifying 
the Java code the Toolkit generated to get LCR up and running. The toolkit was then gradually 
modified to automate the manual changes we had to make in Java. By the Spanning Tree to 
Leader algorithm the Toolkit was completely automated to provide Java code that compiled and 
ran successfully. After that, and after making sure that simple algorithms such as a simple broadcast 
and convergecast could be run, we implemented a more complicated algorithm, the GHS algorithm 
for computing the minimum spanning tree in an arbitrary graph. The successfull implementation 
of this algorithm makes us very confident that the toolkit can be used to implement complex 
distributed systems successfully. 

Performance We now comment on some performance issues: 

1. Scalability: As Figure 3.6 suggests, the Toolkit is scalable. Letting the number of nodes double 
increases the runtime but no more than twice the previous runtime. Moreover, Figure 3.6(e) 
shows that the rate of increase of the running time is smaller as the number of messages 
increases. 

2. Setup time: The time required for MPI to set up all the connections and enable the nodes to 
initialize was not measured in the runtime results. However, when the number of nodes was 
large, this time was also quite significant (around 5-10 minutes). 

3. Resource usage: The generated programs used all the available processing power available to 
them during the whole run. This was mainly because there was no pause between suceessive 
tests for incoming or outgoing messages. 

5 Comments and Suggestions on the Toolkit 

• The NDR Language, used both in the scheduler and the initialization block could be extended 
to support the for v:T in s:Set[T] statement, which is already used in IOA. This would 
make the code much cleaner since using the current syntax, these loops are implemented using 
a while loop and an extra set of two or more variables. 
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LCR Leader Election: lnode / machine 



Broadcast Convergecast: lnode / machine 
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Figure 3.6: (a) to (d) show the time taken to complete the algorithm in regard to the number 
of nodes, (e) shows the time of Broadcast Leader Election in regard to the number of messages 
exchanged. 
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LCR Leader Election: 20 nodes 



Broadcast Convergecast: 16 nodes 
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Figure 3.7: (a) 
machines used. 



to (c) show the time taken to terminate, in regard to the number of physical 
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LCR Leader Election: 4 physical machines 
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Figure 3.8: (a) to (c) show the time taken to terminate, in regard to the number of nodes. 
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Initialization of Map (or Array) entries. The toolkit allows usage of possibly non-initialized 

map/ array entries. As a result, when the compiled code is run, a NullPointerException is 

thrown from the Java environment when these variables are accessed. The following example 

illustrates the problem and demonstrates some possible solutions: 

type Tup = tuple of a : Int , b: Bool 

automaton Test 

signature 

states 

M : Map [Tup] 
initially 

(true =*> SM [1] . a = 3) 
det do 

SM [1] . a := 3 

The user here tries to initialize the tuple before constructing it, which is done using a com- 
mand like sm[i] := [3, true]. One possible solution would be for the toolkit to check for an 
initialization statement of the type sm[i] := . . . in the initially block, before the accessing 
statement (of the type sm[i] .a). If none exists, it could print a warning. Another solution 
could be to arbitrarily initialize all Map / Array Entries. The toolkit already supports such 
kind of initialization, but again if it does not occur, a warning message could be printed. 

In GHS, some parts of the IOA code were used more than once. The code given in [2] makes 
use of a procedure, i.e. a block of statements that can be declared and then called as many 
times as necessary. Given the complexity of GHS, such a feature would be quite useful during 
the design, coding and debugging process. For GHS, there was no need for parameter passing 
to the procedure. Therefore, as a first step, a simple procedure support (without parameter 
passing) from the toolkit would be enough. 

The use of MPI for communication has both advantages and disadvantages. Implementing 
the Toolkit was probably easier using MPI. No low-level communication programming was 
necessary. Furthermore, it has been tested to work for a long time now, and indeed, it works. 
However, it introduces some issues that could have been avoided. Firstly, most of the error 
messages coming from MPI are far from descriptive, (see Troubleshooting $ 4) Moreover, MPI 
sets up a connection between all pairs of nodes, even if these connections are not necessary. 
An n-node LCR needs only n connections, while MPI sets up 0(n 2 ) connections. This is 
probably the reason why on a larger number of nodes, MPI takes up a lot of time to setup. 
We suggest that the possibility of another communication interface, which gives more control 
over these issues (e.g. Java RMI) is examined at some time in the future. 

Finally, we have tested the possibility of pausing between consecutive tests for incoming and 
outgoing messages (MPI's test and Iprobe). Right now a node might be in an infinite loop 
probing for messages for tens of seconds and using up all the processing power available to 
it. Forcing the nodes to sleep for some milliseconds before probing again showed to improve 
performance when the number of nodes per physical machine is large. We have experimented 
with some runtimes of LCR Leader election on 20 nodes on a single machine, using different 
sleep times. The results are shown below: 

27 



Sleep time (milliseconds) 


LCR on 20 nodes, 1 machine runtime (seconds) 


100 


40 


50 


25 


25 


13 


10 


8 


1 


13 
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A Troubleshooting 

1. Sometimes during the composition of the process and mediator automata, two SEND and/or 
two RECEIVE transitions were produced by the composer. This should be fixed before 
generating Java code because the code will not compile later. To prevent this one should make 
sure that the where clauses on the SEND and RECEIVE actions in the process automaton 
match the where clauses for the SendMediator and ReceiveMediator component automata in 
the node composition. 

2. Send/ Receive Convention. In LCR, we have used the convention of [1] for the Send and 
Receive transitions: Send(m, i, j) and Receive(m, j, i). This means that i is always 
the sender and j is always the receiver. This convention sometimes caused problems that were 
hard to debug. We found that using a different convention made it easier to understand and 
debug any problems: Send(m, i, j) and Receive (m, i, j ), meaning that node i sends to 
node j and node i receives from node j respectively. 

3. Currenlty the NDR Language, used in the schedule and initialially blocks does not support 
the for v:T in S:Set[T]. The way to create such loops is to use a while loop of the form: 

uses Cho i s e S e t 



schedule 
states 

tempNbrs : Set [int] , 

k : Int 

do 

tempNbrs := P.nbrs; 

while (^isEmpt y ( t empNbr s ) ) do 

k := cho o s eRandom ( t empNbr s ) ; 

tempNbrs := delete (k, tempNbrs); 

V/o fire action on neighbor k 
od ; 
od 



4. MPI throws a SIGSEV error if the program tries to access a node using a negative number 
as the rank. 

5. Several times when running an algorithm we encountered a j ava . lang . NullPointerException 
during the initialization or the first transitions of the automata. Almost always, this was due 
to a wrong initialization of the state variables in IOA. 

B Bug Reports 

1. In the LCRNode automaton some types such as Status and .States [LCRProcess] were not 
declared automatically. Manual declaration of these types was necessary. The latest version 
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of the toolkit generates these type definitions automatically. [Fixed] 

2. The transitions that connect to the MPI have different signatures, but the implementation of 
the toolkit did not support that. (e.g. Iprobe(i_vl, j_v2) and resp_Iprobe(i_v3, j_v4) 
were not recognized as an MPI pair). We manually changed them to have exactly the same 
signature, for example Iprobe(i_vl, j_v2) and resp_Iprobe(i_vl, j_v2). The latest 
version of the toolkit recognizes them as pairs even if they have different names. [Fixed] 

3. In LCR a statement (const 0) was produced that was not recognized by the code generator. 
We manually changed the IL file produced to (const v999) , and declared the variable v999 to 
something like (v999 zero s8 (scope 1)) where s8 was the sort corresponding to NatSort. 
[Pending] 

4. Formal parameters were not translated in Java. We manually declared the field in the gen- 
erated code, and initialized it to the correct value. The toolkit now translates all formal 
parameters and automatically initializes three special parameters: rank, size (of type Int or 
Nat) and hostName (of type String). [Fixed] 

5. The initialization of the arguments from MPI (the statement MPI . Init(args) ;) had to hap- 
pen in the main() method of the class LCRNode and not ioa.runitime. Automaton. Otherwise, 
the environment would not recognize the correct number of processes running. The toolkit 
has been modified so that this will happen automatically from now on. [Fixed] 

6. The composer tool (incorrectly) removes all the preconditions of internal actions during com- 
position. [Pending] 

7. In the NDR Language translation, no break; statement was produced after a fire block. This 
caused unexpected termination in some algorithms. [Fixed] 

8. IntSort's modulo operator is not compatible with its specification. [Pending] 

9. The syntax of declaring a map is v: Map[Dl, ..., Dn, R] where n > 1. If only one type is 
given, (e.g. mapl: Map [Int]), the checker tool throws a java.lang. InternalError instead of 
a more descriptive Syntax Error message. 
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C Automata Definitions and Input Files 

C.l SendMediator and ReceiveMediator 

These automata were used as the channel between two nodes and the connection of the automata 
to the MPI. 

SendMediator 

type sCall = enumeration of idle, Isend, test 

automaton SendMediator (Msg , Node : Type , i : Node , j : Node ) 

assumes Infinite (Handle ) 

signature 

input SEND(m: Msg, const i, const j) 

output Isend (m: Msg, const i, const j) 

input resp_Isend (handle : Handle , const i, const j) 

output test (handle : Handle , const i, const j) 

input resp_test (flag : Bool , const i, const j) 

states 

status: sCall := idle, 
toSend: Seq[Msg] := {}, 
sent: Seq[Msg] := {}, 
handles: Seq[Handle] := {} 

transitions 

input SEND(m, i, j) 

eff toSend := toSend h m 
output Isend (m , i , j ) 

pre head(toSend) = m ; 

status = idle 
eff toSend := tail ( toSend ) ; 
sent := sent h m ; 
status := Isend 
input resp_Isend (handle , i, j) 

eff handles := handles h handle; 
status := idle 
output test (handle , i, j) 
pre status = idle ; 

handle = head (handles ) 
eff status := test 
input resp_test (f lag , i, j) 
eff if (flag = true) then 

handles := tail (handles ) ; 
sent := tail(sent) 

f i ; 

status := idle 

ReceiveMediator 

type rCall = enumeration of idle , receive , Iprobe 
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automaton ReceiveMediator (Msg , Node: Type, i: Node, j:Node) 

assumes Infinite (Handle ) 

signature 

output RECEIVE (m : Msg , const i, const j) 

output Iprobe (const i, const j) 

input resp_Iprobe (flag : Bool , const i, const j) 

output receive (const i, const j) 

input resp_receive (m : Msg, const i, const j) 

states 

status: rCall := idle, 
toRecv: Seq[Msg] := {}, 
ready: Bool := false 

transitions 

output RECEIVE (m, i, j) 
pre m = head (toRecv) 
eff toRecv := tail(toRecv) 
output Iprobe (i, j) 
pre status = idle ; 

ready = false 
eff status := Iprobe 
input resp_Iprobe ( f lag , i, j) 
eff ready := flag; 
status := idle 
output received, j) 
pre ready = true ; 
status = idle 
eff status := receive 
input resp_receive (m , i, j) 
eff toRecv := toRecv h m ; 
ready := false ; 
status := idle 



C.2 LCR 

Expanded automaton with initialization and scheduling 

uses ChoiceSet ( Int ) 

automaton LCR(rank: Int, size: Int) 
signature 

output receive (N6: Int, N7 : Int) 

where N7 = rank A N6 = mod ( ( rank+size ) -1, size) 
output SEND(m: Int, 12: Int, 13: Int) 

where 13 = mod (rank + 1, size) A 12 = rank 
input resp_Iprobe (f lag : Bool, N4 : Int, N5 : Int) 

where N5 = rank A N4 = mod (( rank+size ) -1, size) 
input resp_test (f lag : Bool, N18 : Int, N19 : Int) 

where N19 = mod (rank + 1, size) A N18 = rank 
input resp_receive (m : Int, N8 : Int, N9 : Int) 

where N9 = rank A N8 = mod (( rank+size ) -1, size) 
input vote 
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output test (handle : Handle, N16 : Int , N17 : Int) 

where N17 = mod(rank + 1, size) A N16 = rank 
output Iprobe(N2: Int, N3 : Int) 

where N3 = rank A N2 = mod ( ( rank+size ) -1, size) 
output RECEIVE (m: Int, 10: Int, II: Int) 

where II = rank A 10 = mod (( rank+size ) -1, size) 
output leader (14: Int) where 14 = rank 
input resp_Isend (handle : Handle, N14 : Int, N15 : Int) 

where N15 = mod(rank + 1, size) A N14 = rank 
output Isend(m: Int, N12 : Int, N13 : Int) 

where N13 = mod(rank + 1, size) A N12 = rank 
states 

P: .States [LCRProcess] , 

RM : Map [Int, _St at es [ReceiveMediat or , Int, Int]], 
SM : Map [Int, .States [SendMediator , Int, Int]] 
initially 

(true => P . pending = {rank} A P . status = idle) 
A 

V j : Int 

((j = mod (( rank + size ) -1, size) 
=> 

RM[j]. status = idle 
A RM [j] . toRecv = {} 
A RM[j]. ready = false) 
A (j = mod ( (rank+size ) -1, size) O- defined(RM, j))) 
A 

V j : Int 

((j = mod(rank + 1, size) 
=> 

SM[j]. status = idle 
A SM[j] . toSend = {> 
A SM[j] . sent = {> 
A SM[j] . handles = {}) 
A (j = mod(rank + 1, size) <^ defined(SM, j))) 
det do 

P . pending := -[rank}; 
P. status := idle; 

RM [mod ( (rank + size ) -1, size)] := [idle, {}, false]; 
SM[mod(rank+l, size)] := [idle, {}, {}, {}] 
od 
transitions 

output receive (N6, N7) 

pre RM [mod (( rank+size ) -1, size)]. ready = true 

A RM [mod (( rank+size ) -1, size )]. status = idle 
eff RM [mod (( rank + size ) -1, size )]. status := receive 
output SEND(m, 12, 13) 

pre P. status 7^ idle A m G (P. pending) 
eff SM[mod(rank + 1, size ) ] . t oSend : = 

(SM[mod(rank + 1, size ) ] . toSend ) h m; 
P . pending := delete(m, P. pending) 
input resp_Iprobe (f lag , N4 , N5) 

eff RM [mod (( rank + size ) -1, size)]. ready := flag; 
RM [mod (( rank+size ) -1, size )]. status := idle 
input resp_test (f lag , N18 , N19) 
eff if flag = true then 

SM[mod(rank + 1, size )]. handles := 
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tail (SM [mod (rank + 1, size )]. handles ) ; 
SM[mod(rank + 1, size)]. sent := 

tail (SM [mod (rank + 1, size)]. sent) 
f i ; 

SM[mod(rank + 1, size )]. status := idle 
input resp_receive (m , N8 , N9) 

eff RM [mod ( ( rank + size ) -1, size ) ] . toRecv : = 

(RM [mod ( ( rank + size ) -1, size ) ] . t oRecv ) h m; 
RM [mod (( rank + size ) -1, size)]. ready := false; 
RM [mod (( rank+size ) -1, size )]. status := idle 
input vote 

eff P. status := voting 
output test (handle, N16 , N17) 

pre SM[mod(rank + 1, size )]. status = idle 

A handle = head (SM [mod ( rank + 1, size )]. handles ) 
eff SM[mod(rank + 1, size )]. status := test 
output Iprobe(N2, N3) 

pre RM [mod (( rank+size ) -1, size )]. status = idle 

A RM [mod (( rank + size ) -1, size)]. ready = false 
eff RM [mod (( rank + size ) -1, size )]. status := Iprobe 
output RECEIVE (m, 10, II) where m > II V m < II V m = II 
pre m = head (RM [mod (( rank + size ) -1, size ) ] . t oRecv ) 
eff if m > II then P . pending := insert (m, P. pending) 
elseif m = II then P. status := elected 
f i ; 
RM [mod (( rank+size ) -1, size ) ] . toRecv := 

tail (RM [mod (( rank+size ) -1, size )]. toRecv ) 
output leader(I4) 

pre P. status = elected 
eff P. status := announced 
input resp_Isend (handle , N14 , N15) 

eff SM[mod(rank + 1, size )]. handles := 

(SM[mod(rank + 1, size )]. handles ) h handle; 
SM[mod(rank + 1, size )]. status := idle 
output Isend(m, N12 , N13) 

pre head (SM [mod (rank + 1, size ) ] . t oSend) = m 

A SM[mod(rank + 1, size )]. status = idle 
eff SM[mod(rank + 1, size ) ] . t oSend := 

tail (SM [mod (rank + 1, size ) ] . t oSend) ; 
SM [mod ( rank + 1, size)]. sent := 

(SM[mod(rank + 1, size)]. sent) h m; 
SM[mod(rank + 1, size )]. status := Isend 
schedule 
do 

fire input vote ; 
while (true) do 
if P . pending 7^ {}■ then 

fire output SEND ( chooseRandom (P . pending ) , rank, mod ( rank+1 , size ) ) fi ; 
if SM [mod (rank+1 , size )]. status = idle 

A SM [mod(rank+l , size )] . toSend / {} then 
fire output 

Isend (head ( SM [mod (rank + 1 , size )]. toSend ) , rank, mod ( rank + 1 , size ) ) fi ; 
if SM [mod (rank+1 , size )]. st atus = idle 

A SM [mod (rank+1 , size )]. handles 7^ -Q then 
fire output 

test (head ( SM [mod ( rank + 1 , size )]. handles ) , rank, mod ( rank + 1 , size ) ) fi ; 
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if RM [mod ( ( rank+size ) -1 , size )]. status = idle 

A RM [mod (( rank+size ) -1 , size )]. ready = false then 
fire output Iprobe (rank , mod ( rank -1 , size ) ) fi ; 
if RM [mod ( (rank + size ) -1 , size )]. status = idle 

A RM [mod (( rank + size ) -1 , size )]. ready = true then 
fire output receive (rank , mod (( rank+size ) -1, size)) f i ; 
if RM [mod ( (rank + size ) -1 , size ) ] . toRecv 7^ {} then 
fire output 
RECEIVE (head (RM [mod (( rank+size ) -1, size)] . toRecv) , 
mod (( rank + size ) -l.size), rank) f i ; 
if P. status = elected then 
fire output leader (rank) fi 
od 
od 

type Status = enumeration of idle, voting, elected, announced 

type rCall = enumeration of idle , receive , Iprobe 

type sCall = enumeration of idle, Isend, test 

type _States [LCRProcess ] = tuple of pending: Set[Int], status: Status 

type .States [ReceiveMediat or , Int , Int] = tuple of status: rCall , toRecv: 

Seq[Int], ready: Bool 
type .States [SendMediat or , Int, Int] = tuple of status: sCall , toSend: 

Seq[Int], sent: Seq[Int], handles: Seq[Handle] 



C.3 Asynchronous Spanning Tree 

Expanded automaton with initialization and scheduling 

type Message = enumeration of search, null 

type rCall = enumeration of idle , receive , Iprobe 

type sCall = enumeration of idle, Isend, test 

type .States [sTreeProcess ] = tuple of parent: Int, reported: 

Bool, send: Map [Int, Message] 
type .States [SendMediat or , Message, Int] = tuple of status: sCall , toSend: 

Seq [Message ] , sent: Seq [Message ] , handles: Seq[Handle] 
type .Stat es [ReceiveMediat or , Message, Int] = tuple of status: rCall , toRecv: 

Seq [Message ] , ready: Bool 

uses ChoiceSet ( Int ) 

automaton sTree(rank: Int, nbrs : Set [Int]) 

signature 

output SEND(m: Message, vlO : Int, vll: Int) where vlO = rank 

output RECEIVE (m : Message, vO : Int, vl : Int) where vO = rank 

output PARENT (j3: Int) 

output test (handle : Handle, vl6 : Int, vl7 : Int) where vl6 = rank 

input resp.test (f lag : Bool, vl8 : Int, vl9 : Int) where vl8 = rank 

output receive (v6: Int, v7 : Int) where v6 = rank 

input resp.receive (m : Message, v8 : Int, v9 : Int) where v8 = rank 

output Iprobe (v2: Int, v3 : Int) where v2 = rank 

input resp.Iprobe (f lag : Bool, v4 : Int, v5 : Int) where v4 = rank 

output Isend (m: Message, vl2 : Int, vl3 : Int) where vl2 = rank 

input resp.Isend (handle : Handle, vl4 : Int, vl5 : Int) where vl4 = rank 

states 

P: .States [sTreeProcess ] , 
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RM : Map[Int, _St ates [ReceiveMediat or , Message, Int]], 
SM : Map [Int, _St ates [SendMediator , Message, Int]], 
°/ Temporary 

tempNbrs : Set [Int] , j : Int 
initially 

true => P . parent = -1 
det do 

P . parent := -1 ; 
P. reported := false; 
tempNbrs : = nbrs ; 
while (^isEmpty (tempNbrs ) ) do 
j := chooseRandom (tempNbrs ) ; 
tempNbrs : = delete (j, tempNbrs); 
if rank = then 

P. send [j] := search 
else 

P.send[j] := null 

f i ; 

RM[j] : = [idle, {}, false]; 
SM[j] : = [idle, {>, {>, {>] 
od 
od 

transitions 

output SEND(m, vlO , vll) where vlO = rank 
pre P.send[vll] = search 

eff SM[vll] . toSend := ( SM [vl 1 ] . toSend ) h m; 
P . send [vll] := null 
output RECEIVE (m, vO , vl) 

pre m = head (RM [vl] . toRecv) 
eff if rank 7^ A P . parent = -1 then 
P . parent := vl ; 
for k: Int in nbrs - {vl} do 

P.send[k] := search 
od 

f i ; 

RM [vl] . toRecv := tail (RM [ vl ] . t oRecv) 
output PARENT (j3) 

pre P . parent = j3 A P. reported = false 
eff P. reported := true 
output receive (v6, v7) 

pre RM[v7]. ready = true A RM [v7] . status = idle 
eff RM [v7] . status := receive 
input resp_receive (m , v6 , v7) 

eff RM [v7] . toRecv := (RM [v7] . toRecv ) h m; 
RM[v7]. ready := false; 
RM [v7] . status := idle 
output test (handle , vl6 , vl7) 

pre SM [vl7] . status = idle A handle = head ( SM [vl7] . handles ) 
eff SM [vl7] . status := test 
input resp_test ( f lag , vl8 , vl9) 
eff if flag = true then 

SM[vl9] . handles := tail (SM [vl9] . handles ) ; 
SM[vl9].sent := tail ( SM [vl9] . sent ) 
f i ; 
SM [vl9] . status := idle 
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output Iprobe(v2, v3) 

pre RM [v3] . status = idle A RM[v3]. ready = false 
eff RM [v3] . status : = Iprobe 
input resp_Iprobe (f lag , v2 , v3) 
eff RM[v3] . ready : = flag; 
RM [v3] . status : = idle 
output Isend(m, vl2 , vl3) 

pre head (SM [vl3] . toSend) = m A SM [vl3] . status = idle 
eff SM[vl3] . toSend := tail ( SM [ vl3] . t oSend) ; 
SM[vl3].sent : = ( SM [vl3] . sent ) h m ; 
SM [vl3] . status := Isend 
input resp_Isend (handle , vl2 , vl3) 

eff SM[vl3] . handles : = ( SM [vl3 ]. handles ) h handle; 
SM [vl3] . status := idle 
schedule 
states 

nb : Set [Int] , 
k : Int 
do 
while (true) do 
nb := nbrs ; 

while (^isEmpty (nb ) ) do 
k := chooseRandom (nb ) ; 
nb := delete(k, nb); 
if P. send [k] = search then 

fire output SEND (search, rank, k) f i ; 
if SM[k]. status = idle A SM [k] . toSend / {} then 

fire output Isend (head (SM [k] . toSend) , rank, k) f i ; 
if SM[k]. status = idle A SM [k] . handles / {} then 

fire output test (head (SM [k] . handles ) , rank, k) f i ; 
if RM [k] . status = idle A RM[k]. ready = false then 

fire output Iprobe (rank, k) fi ; 
if RM [k] . status = idle A RM [k] . ready = true then 

fire output receive (rank , k) f i ; 
if RM[k].toRecv / {} then 

fire output RECEIVE (head (RM [k] . toRecv) , rank, k) f i ; 
if P . parent = k A P. reported = false then 
fire output PARENT (k) fi 
od 
od 
od 



C.4 Asynchronous Broadcast Convergecast 
Expanded automaton with initialization and scheduling 

type Kind = enumeration of beast , ack 

type BCastMsg = tuple of kind : Kind , w : Int 

type Message = union of msg : BCastMsg, kind: Kind 

type rCall = enumeration of idle , receive , Iprobe 

type sCall = enumeration of idle, Isend, test 

type .States [bcastProcess ] = tuple of val : Int , acked : Set[Int], 

parent: Int, reported: 

Bool, send: Map[Int, Seq [Message ]] , temp: BCastMsg 
type _States [SendMediator , Message, Int] = tuple of status: sCall , toSend: 
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Seq [Message ] , sent: Seq [Message ] , handles: Seq[Handle] 
type .States [ReceiveMediat or , Message, Int] = tuple of status: rCall , toRecv: 
Seq [Message ] , ready: Bool 

uses ChoiceSet ( Int ) 

automaton beast (rank: Int, nbrs : Set [Int]) 

assumes bcastNode_Axioms 

assumes P_Axioms 

assumes RM_Axioms 

assumes SM_Axioms 

signature 

output receive (v6: Int, v7 : Int) where v6 = rank 

output SEND(m: Message, vlO : Int, vll: Int) where vlO = rank 

input resp_Iprobe (flag : Bool, v4 : Int, v5 : Int) where v4 = rank 

internal report (v2: Int) where v2 = rank 

input resp_test (f lag : Bool, vl8 : Int, vl9 : Int) where vl8 = rank 

input resp_receive (m : Message, v8 : Int, v9 : Int) where v8 = rank 

output test (handle : Handle, vl6 : Int, vl7 : Int) where vl6 = rank 

output Iprobe(v2: Int, v3 : Int) where v2 = rank 

output RECEIVE (m: Message, vO : Int, vl : Int) where vO = rank 

input resp_Isend (handle : Handle, vl4 : Int, vl5 : Int) where vl4 = rank 

output Isend(m: Message, vl2 : Int, vl3 : Int) where vl2 = rank 

states 

P: _St at es [bcastProcess ] , 

RM : Map [Int , .States [ReceiveMediat or , Message, Int]], 

SM : Map [Int , _States [SendMediator , Message, Int]], 

'/,'/, Temporary variables 

tempNbrs : Set [Int] := {>, 

j : Int 

initially 

true => P . val = 99 
det do 

if rank = then 
P . val := 99; 
(P . temp ). kind := beast; 
(P. temp). w := P . val ; 
tempNbrs := nbrs; 
while (^isEmpty (tempNbrs ) ) do 
j := chooseRandom ( tempNbrs ) ; 
tempNbrs := delete (j, tempNbrs); 
P.send[j] := {} h msg(P.temp) 
od 
else 

P. val := -1; 
tempNbrs := nbrs; 
while (^isEmpty (tempNbrs ) ) do 
j := chooseRandom ( tempNbrs ) ; 
tempNbrs := delete (j, tempNbrs); 
P . send [j] := {} 
od 
f i ; 

P . parent := -1 ; 
P. reported := false; 
P . acked := {}; 
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tempNbrs := nbrs ; 

while (^isEmpty (tempNbrs ) ) do 

j := chooseRandom ( tempNbrs ) ; 

tempNbrs := delete (j, tempNbrs); 

RM[j] : = [idle, {}, false]; 

SM[j] := [idle, {}, {}, {}] 
od 



od 



transitions 

output receive (v6, v7) 

pre RM[v7]. ready = true A RM [v7] . status = idle 
eff RM [v7] . status := receive 
output SEND(m, vlO , vll) where vlO = rank 
pre m = head (P . send [vll] ) 

eff SM[vll] . toSend := ( SM [ vl 1 ] . toSend ) h m; 
P.send[vll] : = tail (P . send [vl 1 ] ) 
input resp_Iprobe (f lag , v4 , v5) 
eff RM[v5]. ready := flag; 
RM [v5] . status := idle 
internal report (v2) 

7,7, The preconditions were (incorrectly) removed 
eff if rank = then P. reported := true 
elseif rank 7^ then 

P . send [P . parent ] := P . send [P . parent ] h kind(ack); 
P. reported := true 
fi 
input resp_test (f lag , vl8 , vl9) 
eff if flag = true then 

SM[vl9] . handles := tail (SM [vl9] . handles ) ; 
SM[vl9] . sent := tail ( SM [vl9] . sent ) 
f i ; 

SM [vl9] . status := idle 
input resp_receive (m , v8 , v9) 

eff RM [v9] . toRecv := (RM [v9] . toRecv ) h m; 
RM[v9]. ready := false; 
RM [v9] . status := idle 
output test (handle , vl6 , vl7) 

pre SM [vl7] . status = idle A handle = head ( SM [vl7] . handles ) 
eff SM [vl7] . status := test 
output Iprobe(v2, v3) 

pre RM [v3] . status = idle A RM[v3]. ready = false 
eff RM [v3] . status := Iprobe 
output RECEIVE (m, vO , vl) 

pre m = head (RM [vl] . toRecv) 

eff if m = kind(ack) then P . acked := (P.acked) U {vl} 
else 

if P.val = -1 then 

P . val := (m.msg).w; 
P . parent := vl ; 

for k: Int in (nbrs) - {vl} do 
P.send[k] := P . send [k] h m 
od 
else P.send[vl] := P.send[vl] h kind(ack) 
fi 
f i ; 
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RM [vl] . toRecv := tail (RM [ vl ] . t oRecv) 
input resp.Isend (handle , vl4 , vl5) 

eff SM [vl5] . handles : = ( SM [vl5 ]. handles ) h handle; 
SM [vl5] . status := idle 
output Isend(m, vl2 , vl3) 

pre head (SM [vl3] . toSend) = m A SM [vl3] . status = idle 
eff SM[vl3] . toSend := tail ( SM [ vl3] . t oSend) ; 
SM[vi3] . sent : = (SM [vl3] . sent ) h m ; 
SM [vl3] . status := Isend 

schedule 
states 

tempNbrs2: Set[Int], 
k : Int 
do 

while (true ) do 
tempNbrs2 := nbrs ; 
while (^isEmpty ( tempNbrs2 ) ) do 
k : = chooseRandom ( tempNbrs2 ) ; 
tempNbrs2 := delete(k, tempNbrs2); 
if P. send[k] / {> then 

fire output SEND (head (P . send [k] ) , rank, k) f i ; 
if SM[k]. status = idle A SM [k] . toSend / {} then 

fire output Isend (head (SM [k] . toSend) , rank, k) f i ; 
if SM[k]. status = idle A SM [k] . handles / {} then 

fire output test (head (SM [k] . handles ) , rank, k) f i ; 
if RM[k]. status = idle A RM [k] . ready = false then 

fire output Iprobe(rank, k) f i ; 
if RM [k] . status = idle A RM [k] . ready = true then 

fire output receive (rank , k) fi ; 
if RM[k].toRecv / {} then 

fire output RECEIVE (head (RM [k] . toRecv) , rank, k) fi 
od; 
if rank = A P . acked = nbrs A P. reported = false then 

fire internal report (rank) f i ; 
if rank 7^ A P . parent 7^ -1 A P . acked = nbrs - {P. parent} A 
P . reported = false then 
fire internal report (rank) fi 
od 
od 



C.5 Leader Election using Asynchronous Broadcast Convergecast 
Expanded automaton with initialization and scheduling 

type Kind = enumeration of beast , ack 

type BCastMsg = tuple of kind: Kind, w: Int 

type AckMsg = tuple of kind : Kind , mx : Int 

type MSG = union of bmsg : BCastMsg, amsg : AckMsg, kind: Kind 

type Message = tuple of msg : MSG, source: Int 

type rCall = enumeration of idle , receive , Iprobe 
type sCall = enumeration of idle, Isend, test 

type .States [bcastLeaderProcess ] = tuple of val : Map [Int, Int], 
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parent: Map[Int, Int], reported: Map[Int, Bool], acked: 
Map[Int, Set[Int]], send: Map[Int, Int, Seq [Message ]] , max: 
Map [Int, Int], elected: Bool, announced: Bool 

type .States [ReceiveMediat or , Message, Int] = tuple of status: rCall , toRecv: 
Seq [Message ] , ready: Bool 

type .States [SendMediat or , Message, Int] = tuple of status: sCall , toSend: 
Seq [Message ] , sent: Seq [Message ] , handles: Seq[Handle] 

type .States [bcastLeaderNode ] = tuple of P: .States [bcastLeaderProcess ] , 
RM : Map[Int, .States [ReceiveMediat or , Message, Int]], SM : 
Map [Int, .States [SendMediator , Message, Int]] 

uses ChoiceSet ( Int ) 

automaton bcastLeader (rank : Int, size : Int , nbrs : Set [Int ] ) 

signature 

output receive (N6: Int, N7 : Int) where N6 = rank 

output SEND(m: Message, N10 : Int, Nil: Int) where N10 = rank 

input resp.Iprobe (f lag : Bool, N4 : Int, N5 : Int) where N4 = rank 

internal finished 

internal report (12: Int, source: Int) where 12 = rank 

output LEADER 

input resp.test (f lag : Bool, N18 : Int, N19 : Int) where N18 = rank 

input resp.receive (m : Message, N8 : Int, N9 : Int) where N8 = rank 

output test (handle : Handle, N16 : Int, N17 : Int) where N16 = rank 

output Iprobe(N2: Int, N3 : Int) where N2 = rank 

output RECEIVE (m: Message, NO: Int, Nl : Int) where NO = rank 

input resp.Isend (handle : Handle, N14 : Int, N15 : Int) where N14 = rank 

output Isend(m: Message, N12 : Int, N13 : Int) where N12 = rank 

states 

P: .States [bcastLeaderProcess ] , 

RM : Map [Int, .States [ReceiveMediat or , Message, Int]], 

SM : Map [Int, .Stat es [SendMediator , Message, Int]], 

tempNbrs : Set [Int], 

c : Int , 

j : Int 

initially 

( true 



V j : Int 
(0 < j 



A j < 16 



(rank = j 



P . val [ j ] = rank 

A (rank / j => P . val [ j ] 
A P . parent [j] = -1 
A P. acked[j] = {} 
A P.max[j] = rank 



•1) 



V k: Int 

(P . send [j , k] 
A k £ nbrs 



o 
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A rank = j 
=> 

P . send [ j , k] 

= {} h [bmsg ( [beast , 991), j]))) 



A P. elected = false) 



V j : Int 

(RM [j] . status = idle 
A RM [j] . toRecv = {} 
A RM[j]. ready = false 
A defined(RM, j)) 
A 

V j : Int 

(SM [j] . status = idle 
A SM[j] . toSend = {} 
A SM[j] . sent = {> 
A SM[j] . handles = {} 
A defined(SM, j)) 
det do 

c := size ; 
while (c > 0) do 
c := c - 1 ; 
if (rank = c) then 

P.val[c] := rank 
else 

P. val [c] := -1 

f i ; 

P.parent[c] := -1; 

P . acked [c] := {}; 

P.max[c] := rank; 

P . reported [c] := false; 

P. elected := false; 

P. announced := false; 

tempNbrs := nbrs ; 

while (^isEmpty (tempNbrs ) ) do 

j := chooseRandom ( tempNbrs ) ; 

tempNbrs := delete (j, tempNbrs); 

P . send [c , j] := {}; 

if c = rank then 

P.send[c, j] := {} h [bmsg ( [beast, 991 ), c] 

fi 
od; 

tempNbrs := nbrs; 
while (^isEmpty (tempNbrs ) ) do 
j := chooseRandom ( tempNbrs ) ; 
tempNbrs := delete (j, tempNbrs); 
RM[j] : = [idle, {}, false]; 
SM[j] := [idle, {}, {}, {}] 
od 
od 
od 

transitions 

output receive (N6, N7) 
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pre RM[N7]. ready = true A RM [N7] . status = idle 
eff RM [N7] . status := receive 
output SEND(m, N10 , Nil) where N10 = rank 
pre m = head (P . send [m . source , Nil]) 
eff SM[N11] . toSend := (SM [Nil] . toSend) h m ; 

P . send [m . source , Nil] := tail (P . send [m . source , Nil]) 
input resp_Iprobe (f lag , N4 , N5) 
eff RM[N5]. ready := flag; 
RM [N5] . status := idle 

output LEADER 

pre P. elected = true A P. announced = false 
eff P. announced := true 

internal finished 

'/. preconditions didn t pass through 
pre P . acked [rank] = nbrs A 

P . reported [rank] = false 
eff P . reported [rank] := true; 

if (P.max[rank] = rank) then 
P. elected := true 

f i ; 
internal report (12, source) 

eff P . send [ source , P . parent [ source ] ] : — 
P . send [ source , P . parent [ source] ] 

h [amsg([ack, P . max [source ]]) , source]; 
P . reported [source ] := true 

input resp_test (f lag , N18 , N19) 
eff if flag = true then 

SM [N19] . handles := tail (SM [N19] . handles ) ; 
SM[N19] . sent := tail (SM [N19] . sent ) 
f i ; 

SM [N19] . status := idle 
input resp_receive (m , N8 , N9) 

eff RM[N9] . toRecv := (RM [N9] . toRecv ) h m ; 
RM[N9]. ready := false; 
RM [N9] . status := idle 
output test (handle, N16 , N17) 

pre SM [N17] . status = idle A handle = head ( SM [N17] . handles ) 
eff SM [N17] . status := test 
output Iprobe(N2, N3) 

pre RM [N3] . status = idle A RM[N3]. ready = false 
eff RM [N3] . status := Iprobe 
output RECEIVE (m, NO, Nl) 

pre m = head (RM [Nl] . toRecv) 
eff if m . msg = kind(ack) then 

P . acked [m . source ] := P . acked [m . sour ce ] U {Nl} 
elseif tag(m.msg) = amsg then 

if P . max [m . sour ce ] < (( (m . msg) . amsg ). mx ) then 

P . max [m . sour ce ] := ( (m . msg) . amsg ). mx 
fi ; 

P . acked [m . source ] := P . acked [m . sour ce ] U {Nl} 
else 

if P . val [m . source] = -1 then 

P . val [m . sour ce ] := ( (m . msg) . bmsg ) . w ; 
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P . parent [m . sour ce ] := Nl ; 

for k: Int in (nbrs) - {Nl} do 

P . send [m . sour ce , k] := P . send [m . sour ce , k] h 
od 
else 

P . send [m . sour ce , Nl] : = 

P . send [m . sour ce , Nl] h [kind(ack), m. source] 
fi 
f i ; 

RM [Nl] . toRecv := tail (RM [Nl] . toRecv) 
input resp_Isend (handle , N14 , N15) 

eff SM [N15] . handles := ( SM [N15 ]. handles ) h handle; 
SM [N15] . status := idle 
output Isend(m, N12 , N13) 

pre head (SM [N13] . toSend) = m A SM [N13] . status = idle 
eff SM[N13] . toSend := tail ( SM [N13] . t oSend) ; 
SM[N13] . sent := (SM [N13] . sent ) h m ; 
SM [N13] . status := Isend 



schedule 
states 

c2 : Int , '/,'/, Source 
tempNbrs2: Set [Int], 
k : Int 

do 

while (true ) do 

c2 := size ; 
while (c2 > 0) do 
c2 := c2 - 1; 
tempNbrs2 := nbrs; 
while (^isEmpty ( tempNbrs2 ) ) do 

k := chooseRandom ( tempNbr s2 ) ; 
tempNbrs2 := delete(k, tempNbrs2); 
if P.send[c2, k] / {} then 

fire output SEND (head (P . send [c2 , k] ) , rank, k) f i ; 
if SM[k]. status = idle A SM[k]. toSend / {} then 

fire output Isend (head (SM [k] . toSend) , rank, k) f i ; 
if SM[k]. status = idle A SM [k] . handles / {} then 

fire output test (head (SM [k] . handles ) , rank, k) f i ; 
if RM [k] . status = idle A RM[k]. ready = false then 

fire output Iprobe(rank, k) fi ; 
if RM [k] . status = idle A RM [k] . ready = true then 

fire output receive (rank , k) f i ; 
if RM[k]. toRecv / {> then 

fire output RECEIVE (head (RM [k] . toRecv) , rank, k) fi 
od; 
if c2 / rank A P. parent [c2] / -1 A 

P.acked[c2] = nbrs - {P . parent [c2] } A 
P . reported [c2] = false then 
fire internal report (rank, c2) f i ; 
if c2 = rank A P . acked [rank] = nbrs A 
P . reported [rank] = false then 
fire internal finished fi ; 
if P. elected = true A P. announced = false then 
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fire output LEADER fi 
od 
od 



od 



C.6 Spanning Tree to Leader Election 

Expanded automaton with initialization and scheduling 



uses ChoiceSet ( Int ) 

automaton sTreeLeader (rank : Int, nbrs : Set [Int]) 
signature 

output receive (N6: Int, N7 : Int) where N6 = rank 
output SEND(m: Message, N10 : Int, Nil: Int) where N10 = rank 
input resp_Iprobe (flag : Bool, N4 : Int, N5 : Int) where N4 = rank 
input resp_test (f lag : Bool, N18 : Int, N19 : Int) where N18 = rank 
input resp_receive (m : Message, N8 : Int, N9 : Int) where N8 = rank 
output test (handle : Handle, N16 : Int, N17 : Int) where N16 = rank 
output Iprobe(N2: Int, N3 : Int) where N2 = rank 
output RECEIVE (m: Message, NO: Int, Nl : Int) where NO = rank 
output leader 

input resp_Isend (handle : Handle, N14 : Int, N15 : Int) where N14 = rank 
output Isend(m: Message, N12 : Int, N13 : Int) where N12 = rank 
states 

P: _St at es [ sTreeLeaderProcess ] , 

RM : Map [Int , .States [ReceiveMediat or , Message, Int]], 
SM : Map [Int, .States [SendMediator , Message, Int]], 
'I, Temporary storage 
j : Int , 

tempNbrs : Set [Int] 
initially 

(true => 

P . receivedElect = {} 
A P . sentElect = {} 
A P. status = idle 
A (size(nbrs) = 1 =>■ 

P . send [chooseRandom (nbrs)] 

= P . send [chooseRandom (nbrs ) ] h elect)) 
A V j : Int 

(j G nbrs => 
(RM [j] . status = idle 
A RM [j] . toRecv = {} 
A RM [j ]. ready = false 
A defined(RM, j))) 
A V j : Int 

(j G nbrs => 
(SM [j] . status = idle 
A SM[j] . toSend = {> 
A SM[j] . sent = {> 
A SM[j] . handles = {} 
A defined(SM, j))) 
det do 

P . receivedElect := {}; 
P. sentElect := {}; 
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P. status := idle; 
tempNbrs : = nbrs ; 
while (^isEmpty (tempNbrs ) ) do 
j := chooseRandom ( tempNbrs ) ; 
tempNbrs : = delete (j, tempNbrs); 
P . send [j] := {>; 
RM[j] := [idle, {}, false]; 
SM[j] := [idle, {>, {>, {}] 
od; 
if size (nbrs) = 1 then 

P . send [chooseRandom (nbrs ) ] := {} h elect 
fi 
od 
transitions 

output receive (N6, N7) 

pre RM[N7]. ready = true A RM [N7] . status = idle 
eff RM [N7] . status := receive 
output SEND(m, N10 , Nil) where N10 = rank 
pre m = head (P . send [Nil] ) 

eff SM[N11] . toSend := (SM [Nil] . toSend) h m; 
P.send[Nll] := tail (P . send [Nl 1 ] ) 
input resp_Iprobe (f lag , N4 , N5) 
eff RM[N5]. ready : = flag; 
RM [N5] . status : = idle 
input resp_test (f lag , N18 , N19) 
eff if flag = true then 

SM [N19] . handles := tail (SM [N19] . handles ) ; 
SM[N19] . sent := tail (SM [N19] . sent ) 
f i ; 

SM [N19] . status := idle 
input resp_receive (m , N8 , N9) 

eff RM[N9] . toRecv := (RM [N9] . toRecv ) h m; 
RM[N9]. ready := false; 
RM [N9] . status := idle 
output test (handle, N16 , N17) 

pre SM [N17] . status = idle A handle = head ( SM [N17] . handles ) 
eff SM [N17] . status := test 
output Iprobe(N2, N3) 

pre RM [N3] . status = idle A RM[N3]. ready = false 
eff RM [N3] . status := Iprobe 
output RECEIVE (m, NO, Nl) 

pre m = head (RM [Nl] . toRecv) 

eff P . receivedElect := insert(Nl, P . receivedElect ) ; 
if size (P . receivedElect ) = size(nbrs) - 1 then 

P . send [ chooseRandom (nbrs - (P . receivedElect )) ] := 

P . send [ chooseRandom (nbrs - (P . receivedElect )) ] h elect; 
P . sentElect := 
insert 

( chooseRandom (nbrs - ( P . receivedElect )) , P. sentElect) 
elseif P . receivedElect = nbrs then 
if Nl G (P . sentElect ) then 

if NO > Nl then P. status := elected fi 
else P. status := elected 
fi 
f i ; 
RM [Nl] . toRecv := tail (RM [Nl ]. toRecv) 
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output leader 

pre P. status = elected 
eff P. status := announced 
input resp_Isend (handle , N14 , N15) 

eff SM [N15] . handles := ( SM [N15 ]. handles ) h handle; 
SM [N15] . status := idle 
output Isend(m, N12 , N13) 

pre head (SM [N13] . toSend) = m A SM [N13] . status = idle 
eff SM[N13] . toSend := tail ( SM [N13] . t oSend) ; 
SM [N13] . sent := ( SM [N13] . sent ) h m ; 
SM [N13] . status := Isend 

schedule 
states 

tempNbrs2 : Set[Int], 
k : Int 
do 
while (true ) do 
tempNbrs2 := nbrs ; 
while (^isEmpty ( tempNbrs2 ) ) do 
k := chooseRandom ( tempNbrs2 ) ; 
tempNbrs2 := delete(k, tempNbrs2); 
if P. send [k] / {} then 

fire output SEND (head (P . send [k] ) , rank, k) f i ; 
if SM[k]. status = idle A SM [k] . toSend / {} then 

fire output Isend (head (SM [k] . toSend) , rank, k) f i ; 
if SM[k]. status = idle A SM [k] . handles / {} then 

fire output test (head (SM [k] . handles ) , rank, k) f i ; 
if RM[k]. status = idle A RM [k] . ready = false then 

fire output Iprobe(rank, k) f i ; 
if RM [k] . status = idle A RM [k] . ready = true then 

fire output receive (rank , k) fi ; 
if RM[k].toRecv / {} then 

fire output RECEIVE (head (RM [k] . toRecv) , rank, k) fi 
od ; 

if P. status = elected then fire output leader fi 
od 
od 

type Status = enumeration of idle, elected, announced 

type Message = enumeration of elect 

type rCall = enumeration of idle , receive , Iprobe 

type sCall = enumeration of idle, Isend, test 

type _States [sTreeLeaderProcess ] = tuple of receivedElect : 

Set [Int], sentElect: Set [Int], status: Status, send: 

Map[Int, Seq [Message] ] 
type .States [ReceiveMediat or , Message, Int] = tuple of status: rCall , 

toRecv : Seq [Message ] , ready: Bool 
type .States [SendMediat or , Message, Int] = tuple of status: sCall , toSend: 

Seq [Message ] , sent: Seq [Message ] , handles: Seq[Handle] 



C.7 GHS 

Expanded automaton with initialization and scheduling 
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uses ChoiceSet (Link ) 

automaton 

GHS(rank: Int , size: Int , links: Set [Link], weight: Map [Link, Int]) 
signature 

input resp_receive (m : Message, N8 : Int, N9 : Int) where N8 = rank 
internal ReceiveChangeRoot ( qp : Link) 

input resp_Iprobe ( f lag : Bool, N4 : Int, N5 : Int) where N4 = rank 
output RECEIVE (m : Message, i: Int, j: Int) where i = rank 
input resp_Isend (handle : Handle, N14 : Int, N15 : Int) where N14 = rank 
output Isend(m: Message, N12 : Int, N13 : Int) where N12 = rank 
internal ReceiveConnect (qp : Link, 1: Int) 
internal ReceiveReport (qp : Link, w: Int) 
internal ReceiveAccept (qp : Link) 

internal Receivelnit iate (qp : Link, 1: Int, c: Null[Edge], st : Status) 
output NotInTree(l: Link) 
output InTree(l: Link) 

output Iprobe(N2: Int, N3 : Int) where N2 = rank 
internal ReceiveTest (qp : Link, 1: Int, c: Null [Edge]) 
input resp_test (f lag : Bool, N18 : Int, N19 : Int) where N18 = rank 
output SEND(m: Message, N10 : Int, Nil: Int) where N10 = rank 
output receive (N6: Int, N7 : Int) where N6 = rank 

output test (handle: Handle, N16 : Int, N17 : Int) where N16 = rank 
internal ReceiveRe j ect (qp : Link) 
input startP 
states 

P: .States [GHSProcess] , 

RM : Map [Int , .States [ReceiveMediat or , Message, Int]], 
SM : Map [Int, _St at es [SendMediator , Message, Int]], 
°/o Temporary 

tempLinks: Set [Link], tempL : Link 
initially 
( true 
=> 

P.nstatus = sleeping 

A P . nf rag = nil 

A P . nlevel = 

A P.bestlink = embed ( chooseRandom ( links ) ) 

A P.bestwt = weight [ chooseRandom ( links ) ] 

A P.testlink = nil 

A P . inbranch = chooseRandom ( links ) 

A P . f indcount = 



A 



V 1 : Link 

(1 G links 



P.lstatus[l] = unknown 

answered [1] = false 
queueOut [1] = {} 
queue In [1] = O)) 



A 


P 


A 


P 


A 


P 



A 

V j : Int 

(RM [j] . status = idle 
A RM [j] . toRecv = {} 
A RM[j]. ready = false 
A defined(RM, j)) 
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A 

V j : Int 

(SM [j] . status = idle 
A SM [j] . toSend = {} 
A SM[j] . sent = {> 
A SM[j] . handles = {} 
A defined(SM, j)) 
det do 

P.nstatus := sleeping; 

P . nf rag := nil ; 

P . nlevel := 0; 

P.bestlink := embed ( chooseRandom ( links )) ; 

P.bestwt := weight [chooseRandom ( links )] ; 

P.testlink := nil; 

P . inbranch : = chooseRandom ( links ) ; 

P.findcount := 0; 

tempLinks := links; 

while (^isEmpty (tempLinks ) ) do 

tempL := chooseRandom ( tempLinks ) ; 
tempLinks := delete ( tempL , tempLinks); 
P . lstatus [tempL] := unknown; 
P . answered [tempL] := false; 
P . queueOut [t empL] := O; 
P . queueln [ [ t empL . t , tempL.s]] := {}■; 
RM[tempL.t] := [idle, {}, false]; 
SM[tempL.t] := [idle, {} , {}, {}] 
od 
od 

transitions 

input resp_receive (m , N8 , N9) 

eff RM[N9] . toRecv := (RM [N9] . t oRecv ) h m; 
RM[N9]. ready := false; 
RM [N9] . status := idle 
internal ReceiveChangeRoot (qp) 

eff P . queueln [qp] := tail (P . queueln [qp] ) ; 

if P . lstatus [(P . bestlink ). val] = branch then 
P . queueOut [ (P . bestlink ). val] : = 

P . queueOut [(P . bestlink) . val] h msg ( CHANGEROOT ) 
else 

P . queueOut [ (P . bestlink ). val] : = 
P . queueOut [(P . bestlink) . val] 

h connMsg ( [CONNECT , P.nlevel]); 
P . lstatus [(P . bestlink ). val] := branch 
fi 
input resp_Iprobe (f lag , N4 , N5) 
eff RM[N5]. ready := flag; 
RM [N5] . status := idle 
output RECEIVE (m, i, j) 

pre m = head (RM [ j ] . toRecv) 

eff P . queueln [ [j , i]] := P . queueln [ [j , i]] h m; 
RM[j]. toRecv := tail (RM [j ]. toRecv) 
input resp_Isend (handle , N14 , N15) 

eff SM[N15] . handles := ( SM [N15 ]. handles ) h handle; 
SM [N15] . status := idle 
output Isend(m, N12 , N13) 
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pre head (SM [N13] . toSend) = m A SM [N13] . status = idle 
eff SM[N13] . toSend := tail ( SM [N13] . t oSend) ; 
SM [N13] . sent := ( SM [N13] . sent ) h m ; 
SM [N13] . status := Isend 
internal ReceiveConnect (qp , 1) 

pre head (P . queueln [qp] ) = connMsg ( [CONNECT , 1]) 
eff P . queueln [qp] := tail (P . queueln [qp] ) ; 
if P.nstatus = sleeping then 

P . minL := embed ( chooseRandom ( links )) ; 
P . min := weight [ (P . minL ). val] ; 
for tempL : Link in links do 

if weight [tempL] < (P. min) then 
P . minL := embed ( tempL ) ; 
P. min := weight [tempL] 
fi 
od ; 
P . lstatus [( P . minL ). val ] := branch; 
P.nstatus := found; 
P . queueOut [ (P . minL ). val ] := 

P . queueOut [(P .minL) . val] h connMsg ( [CONNECT , 0]) 

f i ; 

if 1 < (P.nlevel) then 

P . lstatus [ [qp . t , qp . s ] ] := branch; 
if P.testlink 7^ nil then 

P . queueOut [ [qp . t , qp.s]] := 
P . queueOut [ [qp . t , qp.s]] 

h initMsg ( [INITIATE , P.nlevel, P.nfrag, find]); 
P.findcount := (P . f indcount ) + 1 
else 

P . queueOut [ [qp . t , qp.s]] := 
P . queueOut [ [qp . t , qp.s]] 

h initMsg ( [INITIATE , P.nlevel, P.nfrag, found]) 
fi 
else 

if P . lstatus [ [qp . t , qp.s]] = unknown then 

P . queueln [qp] := P . queueln [qp] h connMsg ( [CONNECT , 1] ) 
else 

P . queueOut [ [qp . t , qp.s]] := 
P . queueOut [ [qp . t , qp.s]] 
h 

initMsg 

([INITIATE , 

(P.nlevel) + 1, 
embed ( [qp . t , qp . s] ) , 
find]) 
fi 
fi 
internal ReceiveReport (qp , w) 

pre head (P . queueln [qp] ) = reportMsg ( [REPORT , w] ) 
eff P . queueln [qp] := tail (P . queueln [qp] ) ; 
if [qp.t, qp.s] 7^ P . inbranch then 
P.findcount := (P.findcount) - 1; 
if w < (P.bestwt) then 
P . bestwt := w ; 
P.bestlink := embed([qp.t, qp.s]) 

f i ; 
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if P.findcount = A P.testlink = nil then 
P.nstatus : = found; 
P . queueOut [P . inbranch ] := 

P. queueOut [P. inbranch] h reportMsg ( [REPORT , P.bestwt]) 
fi 
else 

if P.nstatus = find then 

P . queueln [qp] := P . queueln [qp] h reportMsg ( [REPORT , w] ) 
elseif w > (P.bestwt) then 

if P . lstatus [(P . bestlink ) . val] = branch then 
P . queueOut [ (P . bestlink ). val] : = 

P . queueOut [(P . bestlink) . val] h msg ( CHANGEROOT ) 
else 

P . queueOut [ (P . bestlink ). val] : = 
P . queueOut [ (P . bestlink ) . val] 

h connMsg( [CONNECT , P.nlevel]); 
P . lstatus [ (P . best link ). val ] := branch 
fi 
fi 
fi 
internal ReceiveAccept (qp) 

pre head (P . queueln [qp] ) = msg(ACCEPT) 
eff P . queueln [qp] := tail (P . queueln [qp] ) ; 
P.testlink := nil; 

if weight [ [qp . t , qp.s]] < (P.bestwt) then 
P. bestlink := embed([qp.t, qp.s]); 
P.bestwt :— weight [ [qp . t , qp.s]] 

f i ; 

if P.findcount = A P.testlink = nil then 
P.nstatus := found; 
P . queueOut [P . inbranch] := 

P . queueOut [P . inbranch] h reportMsg ( [REPORT , P.bestwt]) 
fi 
internal Receivelnit iate (qp , 1, c, st ) 

pre head (P . queueln [qp] ) = initMsg ([ INITIATE , 1, c, st]) 
eff P . queueln [qp] := tail (P . queueln [qp] ) ; 
P . nlevel := 1 ; 
P . nf rag := c ; 

if st = find then P.nstatus := find else P.nstatus := found f i ; 
P.S := {}; 
for pr : Link in links do 

if pr . t 7^ qp.s A P . lstatus [pr] = branch then 

P.S := (P.S) U {pr} 
fi 
od; 
for k : Link in P.S do 

P . queueOut [k] := P . queueOut [k] h initMsg ([ INITIATE , 1, c, st]) 
od; 
if st = find then 

P . inbranch := [qp.t, qp.s]; 

P. bestlink := nil; 

P.bestwt := 1000; 

P . minL :— nil ; 

P.min := 1000; 

for tempL : Link in links do 

if weight [tempL] < (P.min) A P . lstatus [tempL] = unknown 
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then 

P . minL := embed ( tempL) ; 
P . min := weight [tempL] 
fi 

od; 
if P . minL 7^ nil then 

P.testlink := P . minL ; 
P . queueOut [ (P . minL ) . val ] : = 
P . queueOut [(P. minL). val] 

h testMsg ( [TEST , P.nlevel, P.nfrag]) 
else 

P.testlink := nil; 

if P.findcount = A P.testlink = nil then 
P.nstatus := found; 
P . queueOut [P . inbranch] := 
P . queueOut [P . inbranch] 

h reportMsg ( [REPORT , P.bestwt]) 
fi 
f i ; 

P.findcount := size(P.S) 
fi 
output HotlnTree(l) 

pre P . answered [1] = false A P.lstatus[l] = rejected 
eff P . answered [1] := true 
output InTree(l) 

pre P . answered [1] = false A P.lstatus[l] = branch 
eff P . answered [1] := true 
output Iprobe(N2, N3) 

pre RM [N3] . status = idle A RM[N3]. ready = false 
eff RM [N3] . status := Iprobe 
internal ReceiveTest (qp , 1, c) 

pre head (P . queueln [qp] ) = testMsg ( [TEST , 1, c]) 
eff P . queueln [qp] := tail (P . queueln [qp] ) ; 
if P.nstatus = sleeping then 

P . minL := embed ( chooseRandom ( links )) ; 
P . min := weight [ (P . minL ). val] ; 
for tempL : Link in links do 

if weight [tempL] < (P. min) then 
P . minL := embed ( tempL ) ; 
P. min := weight [tempL] 
fi 
od ; 
P . lstatus [ (P . minL) . val] := branch; 
P.nstatus := found; 
P . queueOut [ (P . minL ). val ] := 

P . queueOut [(P .minL) . val] h connMsg ( [CONNECT , 0]) 
f i ; 
if 1 > (P.nlevel) then 

P . queueln [qp] := P . queueln [qp] h testMsg ( [TEST , 1, c] ) 
else 

if c / P.nfrag then 

P . queueOut [ [qp . t , qp.s]] := 

P . queueOut [[qp .t , qp.s]] h msg(ACCEPT) 
else 

if P . lstatus [ [qp . t , qp.s]] = unknown then 
P.lstatus[[qp.t, qp.s]] := rejected 
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f i ; 

if P.testlink 7^ embed ( [qp . t , qp.s]) then 
P . queueOut [ [qp . t , qp.s]] := 

P . queueOut [ [qp . t , qp.s]] h msg (REJECT) 
else 

P . minL := nil ; 
P.min := 1000; 
for tempL : Link in links do 
if weight [tempL] < (P.min) 

A P . 1st atus [t empL] = unknown then 
P . minL := embed ( tempL ) ; 
P.min := weight [tempL] 
fi 
od; 
if P . minL 7^ nil then 

P.testlink := P . minL ; 
P . queueOut [ (P . minL ). val ] := 
P . queueOut [(P. minL). val] 

h testMsg ( [TEST , P.nlevel, P.nfrag]) 
else 

P.testlink := nil; 

if P.findcount = A P.testlink = nil then 
P.nstatus := found; 
P . queueOut [P . inbranch] := 
P . queueOut [P . inbranch] 

h reportMsg ( [REPORT , P.bestwt]) 
fi 
fi 
fi 
fi 
fi 
input resp_test (f lag , N18 , N19) 
eff if flag = true then 

SM [N19] . handles := tail (SM [N19] . handles ) ; 
SM[N19] . sent := tail (SM [N19] . sent ) 

f i ; 

SM [N19] . status := idle 
output SEND(m, N10 , Nil) where N10 = rank 
pre m = head (P . queueOut [ [N10 , Nil]]) 
eff SM [Nil] . toSend := ( SM [Nl 1 ] . toSend ) h m; 

P . queueOut [[N10 , Nil]] := tail (P . queueOut [ [N10 , Nil]]) 
output receive (N6, N7) 

pre RM[N7]. ready = true A RM [N7] . status = idle 
eff RM [N7] . status := receive 
output test (handle, N16 , N17) 

pre SM [N17] . status = idle A handle = head ( SM [N17] . handles ) 
eff SM [N17] . status := test 
internal ReceiveRe j ect (qp) 

pre head (P . queueln [qp] ) = msg(REJECT) 
eff P . queueln [qp] := tail (P . queueln [qp] ) ; 

if P . 1st atus [ [qp . t , qp.s]] = unknown then 

P . 1st atus [ [qp . t , qp.s]] := rejected 
f i ; 

P . minL := nil ; 
P.min := 1000; 
for tempL: Link in links do 
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if weight [tempL] < (P.min) A P . 1st atus [tempL] = unknown then 
P . minL := embed ( tempL) ; 
P.min := weight [tempL] 
fi 
od; 
if P . minL 7^ nil then 

P.testlink := P . minL ; 
P . queueOut [ (P . minL ) . val ] : = 
P.queueOut [(P. minL). val] 

h testMsg ( [TEST , P.nlevel, P.nfrag]) 
else 

P.testlink := nil; 

if P.findcount = A P.testlink = nil then 
P.nstatus := found; 
P . queueOut [P . inbranch ] : = 

P . queueOut [P. inbranch] h reportMsg ( [REPORT , P.bestwt]) 
fi 
fi 
input startP 

eff if P.nstatus = sleeping then 

P . minL := embed ( chooseRandom ( links )) ; 
P.min := weight [ (P . minL ). val] ; 
for tempL : Link in links do 

if weight [tempL] < (P.min) then 
P . minL := embed ( tempL ) ; 
P.min := weight [tempL] 
fi 
od; 
P . lstatus [ (P . minL ). val] := branch; 
P.nstatus := found; 
P . queueOut [ (P . minL ). val ] : = 

P. queueOut [(P .minL) . val] h connMsg ( [CONNECT , 0]) 
fi 

schedule 
states 
Inks : Set [Link] , 
Ink : Link 
do 

fire input startP; 
while (true ) do 
Inks := links ; 
while (^isEmpty ( Inks ) ) do 
Ink := chooseRandom ( Inks ) ; 
Inks := delete (Ink, Inks); 
if P . queueOut [Ink] / {} then 

fire output SEND (head (P . queueOut [Ink] ) , rank, lnk.t) fi ; 
if SM [lnk.t] . status = idle A SM [Ink . t ] . t oSend / {} then 

fire output Isend (head (SM [Ink . t] . toSend) , rank, lnk.t) fi ; 
if SM [lnk.t] . status = idle A SM [Ink . t] . handles / {} then 

fire output test (head (SM [Ink . t] . handles ) , rank, lnk.t) fi ; 
if RM [lnk.t] . status = idle A RM [Ink . t] . ready = false then 

fire output Iprobe(rank, lnk.t) f i ; 
if RM [Ink . t] . status = idle A RM [Ink . t] . ready = true then 

fire output receive (rank , lnk.t) fi ; 
if RM [Ink . t] . toRecv / {> then 
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fire output RECEIVE (head (RM [Ink . t] . toRecv) , rank, Ink . t ) fi ; 
if P.queueln [[lnk.t , lnk.s]] / {} A 

tag (head (P . queueln [ [Ink . t , lnk.s]])) = connMsg then 
fire internal ReceiveConnect ( [Ink . t , lnk.s], 
(head (P . queueln [ [Ink . t , Ink . s ] ] ) ) . connMsg . 1) 

f i ; 

if P. queueln [[Ink .t , lnk.s]] / {} A 

tag (head (P . queueln [ [Ink . t , lnk.s]])) = initMsg then 
fire internal Receivelnitiate ( [Ink . t , lnk.s], 
(head(P.queueIn[[lnk.t, lnk.s]])). initMsg. 1, 
(head (P . queueln [ [Ink . t , Ink .s]])). initMsg. c, 
(head(P.queueIn[[lnk.t, lnk.s]])). initMsg. st) 
f i ; 
if P. queueln [[Ink .t , lnk.s]] / {} A 

tag (head (P . queueln [ [Ink . t , lnk.s]])) = testMsg then 
fire internal ReceiveTest ( [Ink . t , lnk.s], 

(head (P . queueln [ [Ink . t , Ink. s]])). testMsg. 1, 
(head(P.queueIn[[lnk.t, lnk.s]])) . testMsg. c) 
f i ; 
if P. queueln [[Ink .t , lnk.s]] / {} A 

head(P . queueln [[Ink . t , lnk.s]]) = msg(ACCEPT) then 
fire internal ReceiveAccept ( [Ink . t , lnk.s]) 
f i ; 
if P. queueln [[Ink .t , lnk.s]] / {} A 

head(P . queueln [[Ink . t , lnk.s]]) = msg(REJECT) then 
fire internal ReceiveRe j ect ( [Ink . t , lnk.s]) 

f i ; 

if P. queueln [[Ink .t , lnk.s]] / {} A 

tag (head (P . queueln [[ Ink . t , lnk.s]])) = reportMsg then 
fire internal ReceiveReport ( [Ink . t , lnk.s], 

(head (P . queueln [ [Ink . t , Ink .s]])) . reportMsg. w) 
f i ; 
if P. queueln [[Ink .t , lnk.s]] / {} A 

head (P. queueln [[Ink . t , lnk.s]]) = msg ( CHANGEROOT ) then 
fire internal ReceiveChangeRoot ( [Ink . t , lnk.s]) 

f i ; 

if P . answered [Ink] = false A P . 1st atus [Ink] = branch then 

fire output InTree(lnk) fi ; 
if P . answered [Ink] = false A P . 1st atus [Ink] = rejected then 
fire output NotlnTree ( Ink) fi 
od 
od 
od 

type Nstatus = enumeration of sleeping, find, found 

type Edge = tuple of s : Int , t : Int 

type Link = tuple of s : Int , t : Int 

type Lstatus = enumeration of unknown, branch, rejected 

type Msg = enumeration of CONNECT, INITIATE, TEST, REPORT, ACCEPT, REJECT, 

CHANGEROOT 

type ConnMsg = tuple of msg : Msg , 1 : Int 

type Status = enumeration of find, found 

type InitMsg = tuple of msg: Msg, 1: Int, c: Null [Edge], st : Status 

type TestMsg = tuple of msg: Msg, 1: Int, c: Null [Edge] 

type ReportMsg = tuple of msg : Msg , w : Int 

type Message = union of connMsg: ConnMsg, initMsg: InitMsg, testMsg: 
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TestMsg , reportMsg : ReportMsg , msg : Msg 

type rCall = enumeration of idle , receive , Iprobe 

type sCall = enumeration of idle, Isend, test 

type .States [GHSProcess ] = tuple of nstatus : Nstatus, nf rag : Null [Edge], 
nlevel : Int , bestlink: Null [Link], bestwt: Int , testlink: Null [Link], 
inbranch : Link, findcount: Int, lstatus: Map [Link, Lstatus], queueOut : 
Map [Link , Seq [Message ]] , queueln: Map [Link , Seq [Message ]] , answered: 
Map [Link , Bool], min : Int, minL : Null [Link], S: Set [Link] 

type .States [ReceiveMediat or , Message, Int] = tuple of status: rCall , 
toRecv : Seq [Message ] , ready: Bool 

type .States [SendMediat or , Message, Int] = tuple of status: sCall , toSend: 
Seq [Message ] , sent: Seq [Message ] , handles: Seq[Handle] 
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Table D.l: Runtime results of the LCR Leader Election Algorithm 
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Table D.2: Runtime results of the Asynchronous Spanning Tree algorithm 
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Table D.3: Runtime results of the Asynchronous Broadcast Convergecast algorithm 
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Table D.4: Runtime results of the Leader Election using the Asynchronous Broadcast Convergecast 
Algorithm 
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Table D.5: Runtime results of the Spanning Tree to Leader Election Algorithm 
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Table D.6: Runtime results of the GHS Minimum Spanning Tree algorithm 
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E Traces of runs 

E.l LCR on 8 nodes 
Complete trace 



Initialization starts (0) on loon.csail.mit.edu at 7:25:29:615 

Modified state variables: 

P — > [pending: (), status: idle] 

RM -» Map{> 

SM -> Map{> 

rank — > null 

size — > null 

Initialization ends 

transition: input vote () in automaton LCR(O) 

on loon.csail.mit.edu at 7:25:29:625 
Modified state variables: 
P — > [pending: (0), status: voting] 

RM — > Map{[7 -> [status: idle, toRecv: {}, ready: false]] } 
SM — > Map{[l -> [status: idle, toSend: {}, sent: {} , handles: {}]] > 
rank — > 
size — > 8 

transition: output SEND(0, 0, 1) in automaton LCR(O) 

on loon.csail.mit.edu at 7:25:29:629 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[1 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {0 } Elements removed: {}] }] } 

Initialization starts (2) on parrot.csail.mit.edu at 7:25:29:820 

Modified state variables: 

P — > [pending: (), status: idle] 

RM -> Map{} 

SM -> Map{} 

rank — > null 

size — > null 

Initialization ends 

Initialization starts (3) on tui.csail.mit.edu at 7:25:29:927 

transition: input vote () in automaton LCR (2) 

on parrot.csail.mit.edu at 7:25:29:960 
Modified state variables: 
P — > [pending: (2), status: voting] 

RM — » Map{[l -> [status: idle, toRecv: {}, ready: false]] } 
SM — > Map{[3 -> [status: idle, toSend: {}, sent: {}, handles: {}]] > 
rank — > 2 
size — > 8 

Modified state variables: 

P — > [pending: (), status: idle] 

RM -» Map{> 

SM -> Map{} 

rank — » null 

size — » null 

Initialization ends 

Initialization starts (7) on tui.csail.mit.edu at 7:25:30:169 
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transition: output SEND (2, 2, 3) in automaton LCR(2) 

on parrot.csail.mit.edu at 7:25:30:200 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[3 -> Tuple, modified fields: {[toSend 
Sequence, elements added: {2 } Elements removed: O] }] } 

Modified state variables: 

P — > [pending: (), status: idle] 

RM -> Map{> 

SM -> Map{> 

rank — > null 

size — > null 

Initialization ends 

Initialization starts (6) on parrot.csail.mit.edu at 7:25:30:247 

Modified state variables: 

P — > [pending: (), status: idle] 

RM -> Map{} 

SM -> Map{} 

rank — > null 

size — > null 

Initialization ends 

Initialization starts (5) on condor.csail.mit.edu at 7:25:30:286 

transition: input vote () in automaton LCR(6) 

on parrot.csail.mit.edu at 7:25:30:313 
Modified state variables: 
P — > [pending: (6), status: voting] 

RM — > Map{[5 -> [status: idle, toRecv: {}, ready: false]] } 
SM — > Map{[7 -> [status: idle, toSend: {} , sent: {} , handles: O] ] } 
rank — > 6 
size — > 8 

Modified state variables: 

P — > [pending: (), status: idle] 

RM -» Map{} 

SM -> Map{} 

rank — > null 

size — > null 

Initialization ends 

transition: input vote () in automaton LCR(5) 

on condor.csail.mit.edu at 7:25:30:426 
Modified state variables: 
P — > [pending: (5), status: voting] 

RM — > Map{[4 -> [status: idle, toRecv: {}, ready: false]] } 
SM — > Map{[6 -> [status: idle, toSend: {>, sent: {} , handles: {}]] > 
rank — > 5 
size — > 8 

transition: input vote () in automaton LCR(3) 

on tui.csail.mit.edu at 7:25:30:460 
Modified state variables: 
P — » [pending: (3), status: voting] 

RM — » Map{[2 -> [status: idle, toRecv: {}, ready: false]] } 
SM — » Map{[4 -> [status: idle, toSend: {} , sent: {} , handles: {}]] > 
rank — » 3 
size — > 8 
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transition: output SEND (6, 6, 7) in automaton LCR(6) 

on parrot.csail.mit.edu at 7:25:30:497 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[7 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {6 } Elements removed: {}] }] } 

transition: input vote () in automaton LCR(7) 

on tui.csail.mit.edu at 7:25:30:710 
Modified state variables: 
P — > [pending: (7), status: voting] 

RM — > Map{[6 -> [status: idle, toRecv: {}, ready: false]] } 
SM — > Map-[[0 -> [status: idle, toSend: {}, sent: {}, handles: {}]] > 
rank — > 7 
size — > 8 

transition: output SEND (5, 5, 6) in automaton LCR(5) 

on condor.csail.mit.edu at 7:25:30:737 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[6 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {5 } Elements removed: {}] }] } 

transition: output SEND (3, 3, 4) in automaton LCR(3) 

on tui.csail.mit.edu at 7:25:30:959 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[4 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {3 } Elements removed: {}] }] } 

transition: output SEND (7, 7, 0) in automaton LCR(7) 

on tui.csail.mit.edu at 7:25:30:989 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[0 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {7 } Elements removed: {}] }] } 

Initialization starts (1) on condor.csail.mit.edu at 7:25:31:086 

Modified state variables: 

P — > [pending: (), status: idle] 

RM -> Map{> 

SM -► Map{> 

rank — > null 

size — > null 

Initialization ends 

transition: input vote () in automaton LCR(l) 

on condor.csail.mit.edu at 7:25:31:347 
Modified state variables: 
P — > [pending: (1), status: voting] 

RM — > Map{[0 -> [status: idle, toRecv: {}, ready: false]] } 
SM — > Map{[2 -> [status: idle, toSend: {} , sent: {>, handles: {}]] } 
rank — > 1 
size — > 8 

transition: output RECEIVE (7, 7, 0) in automaton LCR(O) 
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on loon.csail.mit.edu at 7:25:31:445 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> (7)] } 
SM — » Map, modified entries: {[1 -> Tuple, modified fields: {[toSend -> 

Sequence, elements added: {}■ Elements removed: {0 }] [sent -> Sequence, elements 

added: {0 } Elements removed: {}] }] } 

transition: output SEND (7, 0, 1) in automaton LCR(O) 

on loon.csail.mit.edu at 7:25:31:448 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — » Map, modified entries: {[1 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {7 } Elements removed: {}] }] } 

transition: output SEND(1, 1, 2) in automaton LCR(l) 

on condor.csail.mit.edu at 7:25:31:489 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[2 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {1 } Elements removed: {}] }] } 



transition: output RECEIVE (0, 0, 1) in automaton LCR(l) 

on condor.csail.mit.edu at 7:25:32:205 
Modified state variables: 
SM — > Map, modified entries: {[2 -> Tuple, modified fields: {[toSend -> 

Sequence, elements added: {} Elements removed: {1 }] [sent -> Sequence 

added: {1 } Elements removed: {}■] }] } 



elements 



transition: output RECEIVE (1, 1, 2) in automaton LCR(2) 

on parrot.csail.mit.edu at 7:25:32:243 
Modified state variables: 
SM — > Map, modified entries: {[3 -> Tuple, modified fields: {[toSend -> 

Sequence, elements added: {} Elements removed: {2 }] [sent -> Sequence 

added: {2 } Elements removed: {}■] }] } 



elements 



transition: output RECEIVE (6, 6, 7) in automaton LCR(7) 

on tui.csail.mit.edu at 7:25:32:274 
Modified state variables: 
SM — » Map, modified entries: {[0 -> Tuple, modified fields: {[toSend -> 

Sequence, elements added: {} Elements removed: {7 }] [sent -> Sequence, elements 

added: {7 } Elements removed: {}■] }] } 

transition: output RECEIVE (7, 0, 1) in automaton LCR(l) 

on condor.csail.mit.edu at 7:25:32:325 
Modified state variables: 
P — > Tuple, modified fields: {[pending -> (7)] } 

transition: output RECEIVE (5, 5, 6) in automaton LCR(6) 

on parrot.csail.mit.edu at 7:25:32:437 
transition: output SEND (7, 1, 2) in automaton LCR(l) 

on condor.csail.mit.edu at 7:25:32:466 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[2 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {7 } Elements removed: {}] }] } 
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Modified state variables: 

SM — > Map, modified entries: {[7 -> Tuple, modified fields: {[toSend -> 

Sequence, elements added: {} Elements removed: {6 }] [sent -> Sequence, elements 

added: {6 } Elements removed: {}] ]-] }■ 

transition: output RECEIVE (2, 2, 3) in automaton LCR(3) 

on tui.csail.mit.edu at 7:25:32:584 
Modified state variables: 
SM — » Map, modified entries: {[4 -> Tuple, modified fields: {[toSend -> 

Sequence, elements added: {]■ Elements removed: {3 }] [sent -> Sequence, elements 

added: {3 } Elements removed: {}] }] } 

transition: output RECEIVE (7, 1, 2) in automaton LCR(2) 

on parrot.csail.mit.edu at 7:25:32:777 
Modified state variables: 
P — > Tuple, modified fields: {[pending -> (7)] } 

transition: output SEND (7, 2, 3) in automaton LCR(2) 

on parrot.csail.mit.edu at 7:25:32:997 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[3 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {7 } Elements removed: {}] }] } 

transition: output RECEIVE (7, 2, 3) in automaton LCR(3) 

on tui.csail.mit.edu at 7:25:33:121 
Modified state variables: 
P — > Tuple, modified fields: {[pending -> (7)] } 

transition: output SEND (7, 3, 4) in automaton LCR(3) 

on tui.csail.mit.edu at 7:25:33:135 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[4 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {7 } Elements removed: {}] }] } 

Initialization starts (4) on loon.csail.mit.edu at 7:25:36:355 

Modified state variables: 

P — > [pending: (), status: idle] 

RM -> Map{} 

SM -> Map{} 

rank — > null 

size — > null 

Initialization ends 

transition: input vote () in automaton LCR(4) 

on loon.csail.mit.edu at 7:25:36:365 
Modified state variables: 
P — > [pending: (4), status: voting] 

RM — > Map{[3 -> [status: idle, toRecv: {}, ready: false]] } 
SM — > Map{[5 -> [status: idle, toSend: {}, sent: {>, handles: {}]] > 
rank — > 4 
size — > 8 

transition: output SEND (4, 4, 5) in automaton LCR(4) 

on loon.csail.mit.edu at 7:25:36:369 
Modified state variables: 



63 



P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[5 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {4 } Elements removed: {}] }] } 

transition: output RECEIVE (3, 3, 4) in automaton LCR(4) 
on loon.csail.mit.edu at 7:25:36:553 

Modified state variables: 

SM — > Map, modified entries: {[5 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {}■ Elements removed: {4 }] [sent -> Sequence, elements 
added: {4 } Elements removed: {}] }] } 

transition: output RECEIVE (7, 3, 4) in automaton LCR(4) 

on loon.csail.mit.edu at 7:25:36:557 
Modified state variables: 
P — > Tuple, modified fields: {[pending -> (7)] } 

transition: output SEND (7, 4, 5) in automaton LCR(4) 

on loon.csail.mit.edu at 7:25:36:559 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[5 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {7 } Elements removed: {}] }] } 

transition: output RECEIVE (4, 4, 5) in automaton LCR(5) 

on condor.csail.mit.edu at 7:25:37:074 
Modified state variables: 
SM — > Map, modified entries: {[6 -> Tuple, modified fields: {[toSend -> 

Sequence, elements added: {} Elements removed: {5 }] [sent -> Sequence, elements 

added: {5 } Elements removed: {}] }] } 

transition: output RECEIVE (7, 4, 5) in automaton LCR(5) 

on condor.csail.mit.edu at 7:25:37:274 
Modified state variables: 
P — > Tuple, modified fields: {[pending -> (7)] } 

transition: output SEND (7, 5, 6) in automaton LCR(5) 

on condor.csail.mit.edu at 7:25:37:280 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[6 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {7 } Elements removed: {}] }] } 

transition: output RECEIVE (7, 5, 6) in automaton LCR(6) 

on parrot.csail.mit.edu at 7:25:37:755 
Modified state variables: 
P — > Tuple, modified fields: {[pending -> (7)] } 

transition: output SEND (7, 6, 7) in automaton LCR(6) 

on parrot.csail.mit.edu at 7:25:37:770 
Modified state variables: 

P — > Tuple, modified fields: {[pending -> ()] } 

SM — > Map, modified entries: {[7 -> Tuple, modified fields: {[toSend -> 
Sequence, elements added: {7 } Elements removed: {}] }] } 

transition: output RECEIVE (7, 6, 7) in automaton LCR(7) 
on tui.csail.mit.edu at 7:25:37:872 
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Modified state variables: 

P — > Tuple, modified fields: {[status -> elected] } 

transition: output leader (7) in automaton LCR(7) 

on tui.csail.mit.edu at 7:25:37:874 
Modified state variables: 

P — > Tuple, modified fields: {[status -> announced] } 



E.2 Asynchronous Spanning Tree on 16 nodes 
Complete trace 



Begin initialization 

Modified state variables: 

P — > [nbrs : (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — » ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

Begin initialization 

Modified state variables: 

P — > [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

transition: input initialize () in automaton sTreeNode (6) on 
tui . csail .mit . edu 

Modified state variables: 

P — » [nbrs: (10 2 5 7), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runtime . adt . MapSort 

i -> 6 

Begin initialization 

Modified state variables: 

P — > [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

transition: input initialize () in automaton sTreeNode (0) on 
loon . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (1 4), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i -> 

transition: output SEND ( search , 0, 4) in automaton sTreeNode (0) on 
loon . csail .mit . edu 

Modified state variables: 
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P — > [nbrs : (1 4), parent: -1, reported: false, send: 
ioa . runtime . adt . Map Sort] 

SM — > ioa . runtime . adt . MapSort 

Begin initialization 

Modified state variables: 

P — » [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

transition: output SEND ( search , 0, 1) in automaton sTreeNode (0) on 
loon . csail . mit . edu 

Modified state variables: 

P — > [nbrs: (1 4), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: input initialize () in automaton sTreeNode (11) on 
parrot . csail .mit . edu 

Begin initialization 

Modified state variables: 
Modified state variables: 

P — > [nbrs: (10 15 7), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 
i -> 11 

P — > [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

Begin initialization 

Modified state variables: 

P — > [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

transition: input initialize () in automaton sTreeNode (12) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (13 8), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — ► ioa . runt ime . adt . MapSort 

SM — > ioa . runtime . adt . MapSort 

i -> 12 

transition: input initialize () in automaton sTreeNode (10) on 
nene . csail .mit . edu 

transition: input initialize () in automaton sTreeNode (5) on 
parrot . csail .mit . edu 

Begin initialization Begin initialization 
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Modified state variables: 

P — > [nbrs : (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

transition: input initialize () in automaton sTreeNode(l) on 
blackbird . csail .mit . edu 

Begin initialization 

Modified state variables: 

P — > [nbrs: (0 2 5), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

Modified state variables: 

P — » [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

Begin initialization 

Modified state variables: 

P — > [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

RM — > ioa . runtime . adt . MapSort Begin initialization 

Modified state variables: 

P — > [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

SM — > ioa . runt ime . adt . MapSort 

i -> 1 



Modified state variables: 

P — > [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

Begin initialization 

transition: input initialize () in automaton sTreeNode (4) on 
parrot . csail .mit . edu 

transition: input initialize () in automaton sTreeNode (7) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

Modified state variables: 

P — > [nbrs: (11 3 6), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 



67 



SM — > ioa . runt ime . adt . MapSort 

i -> 7 

transition: input initialize () in automaton sTreeNode (13) on 
tui . csail .mit . edu 

transition: input initialize () in automaton sTreeNode (9) on 
condor . csail . mit . edu 

Modified state variables: 

P — > [nbrs : (12 14 9), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i -> 13 

Begin initialization 

Modified state variables: 
Modified state variables: 

P — > [nbrs: (10 13 5 8), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

P — > [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

SM — > ioa . runt ime . adt . MapSort 

RM — > ioa . runt ime . adt . MapSort 

i — > 9 SM — > ioa . runt ime . adt . MapSort 

i — ► null 

End initialization 

transition: input initialize () in automaton sTreeNode (15) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (11 14), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runtime . adt . MapSort 

i -> 15 

Modified state variables: 

P — > [nbrs: (14 6 9), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 
i -> 5 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

transition: input initialize () in automaton sTreeNode (14) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 13 15), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 
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i -> 14 

Modified state variables: 

P — > [nbrs : (0 5 8), parent: -1, reported: false, send: 
ioa . runtime . adt . Map Sort] 

RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 
i -> 4 

Begin initialization 

Modified state variables: 

P — > [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

transition: input initialize () in automaton sTreeNode (8) on 
blackbird . csail .mit . edu 

transition: output RECEIVE ( search , 4, 0) in automaton sTreeNode (4) on 
parrot . csail .mit . edu 

transition: output RECEIVE ( search , 1, 0) in automaton sTreeNode (1) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (0 2 5), parent: 0, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runtime . adt . MapSort 
Modified state variables: 

P — > [nbrs: (0 5 8), parent: 0, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

transition: output PARENT (0) in automaton sTreeNode (4) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (0 5 8), parent: 0, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output PARENT (0) in automaton sTreeNode (1) on 
blackbird . csail .mit . edu 

Modified state variables: 



transition: output SEND(search, 4, 8) in automaton sTreeNode (4) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (0 2 5), parent: 0, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND (search, 1, 2) in automaton sTreeNode (1) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (0 2 5), parent: 0, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — > ioa . runt ime . adt . MapSort 

P — > [nbrs: (0 5 8), parent: 0, reported: true, send: 
ioa . runtime . adt . MapSort] 
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SM — > ioa . runt ime . adt . MapSort 

Modified state variables: 

P — > [nbrs : (12 4 9), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 
i -> 8 

transition: output SEND ( search , 1, 5) in automaton sTreeNode(l) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (0 2 5), parent: 0, reported: true, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND ( search , 4, 5) in automaton sTreeNode (4) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (0 5 8), parent: 0, reported: true, send: 
ioa . runtime . adt . MapSort] 

RM — » ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 5, 1) in automaton sTreeNode (5) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (14 6 9), parent: 1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — » ioa . runt ime . adt . MapSort 

transition: output PARENT (1) in automaton sTreeNode (5) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (14 6 9), parent: 1, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND (search, 5, 4) in automaton sTreeNode (5) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (14 6 9), parent: 1, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — > ioa . runt ime . adt . MapSort 

Modified state variables: 

P — > [nbrs: (11 14 6 9), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 
i -> 10 

transition: output RECEIVE ( search , 5, 4) in automaton sTreeNode (5) on 
parrot . csail .mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 
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SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (search, 5, 9) in automaton sTreeNode (5) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs : (14 6 9), parent: 1, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — * ioa . runt ime . adt . MapSort 

transition: output SEND ( search , 5, 6) in automaton sTreeNode (5) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (14 6 9), parent: 1, reported: true, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 9, 5) in automaton sTreeNode (9) on 
condor . csail . mit . edu 

Modified state variables: 

P — > [nbrs: (10 13 5 8), parent: 5, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

transition: output PARENT (5) in automaton sTreeNode (9) on 
condor . csail . mit . edu 

Modified state variables: 

P — > [nbrs: (10 13 5 8), parent: 5, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND (search, 9, 13) in automaton sTreeNode (9) on 
condor . csail . mit . edu 

Modified state variables: 

P — > [nbrs: (10 13 5 8), parent: 5, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (search, 9, 8) in automaton sTreeNode (9) on 
condor . csail . mit . edu 

Modified state variables: 

P — > [nbrs: (10 13 5 8), parent: 5, reported: true, send: 
ioa . runtime . adt . MapSort] 

RM — » ioa . runt ime . adt . MapSort 

SM — » ioa . runt ime . adt . MapSort 

transition: output SEND (search, 9, 10) in automaton sTreeNode (9) on 
condor . csail . mit . edu 

Modified state variables: 

P — > [nbrs: (10 13 5 8), parent: 5, reported: true, send: 
ioa . runtime . adt . MapSort] 

RM — » ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 8, 4) in automaton sTreeNode (8) on 
blackbird . csail .mit . edu 

transition: output RECEIVE ( search , 4, 5) in automaton sTreeNode (4) on 
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parrot . csail .mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

Modified state variables: 

P — > [nbrs : (12 4 9), parent: 4, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

transition: output PARENT (4) in automaton sTreeNode (8) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — » [nbrs: (12 4 9), parent: 4, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND ( search , 8, 9) in automaton sTreeNode (8) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (12 4 9), parent: 4, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 8, 9) in automaton sTreeNode (8) on 
blackbird . csail .mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (search, 8, 12) in automaton sTreeNode (8) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (12 4 9), parent: 4, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 9, 8) in automaton sTreeNode (9) on 
condor . csail .mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 10, 9) in automaton sTreeNode (10) on 
nene . csail .mit . edu 

Modified state variables: 

P — ► [nbrs: (11 14 6 9), parent: 9, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — ► ioa . runt ime . adt . MapSort 

transition: output PARENT (9) in automaton sTreeNode (10) on 
nene . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (11 14 6 9), parent: 9, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND (search, 10, 6) in automaton sTreeNode (10) on 
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nene . csail .mit . edu 

Modified state variables: 

P — > [nbrs : (11 14 6 9), parent: 9, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND ( search , 10, 11) in automaton sTreeNode ( 10) on 
nene . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (11 14 6 9), parent: 9, reported: true, send: 
ioa . runtime . adt . MapSort] 

RM — » ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND ( search , 10, 14) in automaton sTreeNode ( 10) on 
nene . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (11 14 6 9), parent: 9, reported: true, send: 
ioa . runtime . adt . MapSort] 

RM — » ioa . runt ime . adt . MapSort 

SM — » ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 6, 5) in automaton sTreeNode (6) on 
tui . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 2 5 7), parent: 5, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

transition: output PARENT (5) in automaton sTreeNode (6) on tui.csail.mit.edu 
Modified state variables: 

P — > [nbrs: (10 2 5 7), parent: 5, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND (search, 6, 2) in automaton sTreeNode (6) on 
tui . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 2 5 7), parent: 5, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (search, 6, 7) in automaton sTreeNode (6) on 
tui . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 2 5 7), parent: 5, reported: true, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (search, 6, 10) in automaton sTreeNode (6) on 
tui . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 2 5 7), parent: 5, reported: true, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 
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transition: output RECEIVE ( search , 6, 10) in automaton sTreeNode (6) on 
tui . csail .mit . edu 

transition: output RECEIVE ( search , 10, 6) in automaton sTreeNode (10) on 
nene . csail .mit . edu 

Modified state variables: 

Modified state variables: RM — > ioa . runt ime . adt . MapSort 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort SM — > ioa . runt ime . adt . MapSort 



transition: output RECEIVE ( search , 13, 9) in automaton sTreeNode (13) on 
tui . csail .mit . edu 

Modified state variables: 

P — > [nbrs : (12 14 9), parent: 9, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

transition: output PARENT (9) in automaton sTreeNode (13) on 
tui. csail. mit. edu 

Modified state variables: 

P — > [nbrs: (12 14 9), parent: 9, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND(search, 13, 14) in automaton sTreeNode ( 13) on 
tui . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (12 14 9), parent: 9, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 12, 8) in automaton sTreeNode (12) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (13 8), parent: 8, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

transition: output PARENT (8) in automaton sTreeNode ( 12) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (13 8), parent: 8, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND(search, 12, 13) in automaton sTreeNode ( 12) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (13 8), parent: 8, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — » ioa . runt ime . adt . MapSort 

transition: output SEND(search, 13, 12) in automaton sTreeNode (13) on 
tui . csail .mit . edu 

Modified state variables: 
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P — > [nbrs : (12 14 9), parent: 9, reported: true, send: 
ioa . runtime . adt . Map Sort] 

RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 13, 12) in automaton sTreeNode ( 13) on 
tui . csail .mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 



transition: output RECEIVE ( search , 12, 13) in automaton sTreeNode ( 12) on 
parrot . csail .mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 11, 10) in automaton sTreeNode ( 11) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 15 7), parent: 10, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

transition: output PARENT (10) in automaton sTreeNode ( 1 1) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 15 7), parent: 10, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND (search, 11, 15) in automaton sTreeNode ( 1 1) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 15 7), parent: 10, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — > ioa . runtime . adt . MapSort 

transition: output RECEIVE ( search , 7, 6) in automaton sTreeNode (7) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (11 3 6), parent: 6, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

transition: output PARENT (6) in automaton sTreeNode (7) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (11 3 6), parent: 6, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND (search, 7, 3) in automaton sTreeNode (7) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (11 3 6), parent: 6, reported: true, send: 
ioa . runtime . adt . MapSort] 
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SM — > ioa . runt ime . adt . MapSort 

transition: output SEND ( search , 7, 11) in automaton sTreeNode (7) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs : (11 3 6), parent: 6, reported: true, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

Begin initialization 

Modified state variables: 

P — > [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — > null 

End initialization 

transition: input initialize () in automaton sTreeNode (3) on 
nene . csail .mit . edu 

transition: output SEND(search, 11, 7) in automaton sTreeNode (11) on 
parrot . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 15 7), parent: 10, reported: true, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runtime . adt . MapSort 

transition: output RECEIVE ( search , 11, 7) in automaton sTreeNode (11) on 
parrot . csail .mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 7, 11) in automaton sTreeNode (7) on 
blackbird . csail .mit . edu 

Modified state variables: 

RM — > ioa . runtime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 14, 10) in automaton sTreeNode ( 14) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 13 15), parent: 10, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

transition: output PARENT (10) in automaton sTreeNode ( 14) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 13 15), parent: 10, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND(search, 14, 15) in automaton sTreeNode ( 14) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 13 15), parent: 10, reported: true, send: 
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ioa . runtime . adt . Map Sort] 

SM — > ioa . runtime . adt . MapSort 

Modified state variables: 

P — > [nbrs : (2 7), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — » ioa . runtime . adt . MapSort 
SM — > ioa . runtime . adt . MapSort 
i -> 3 

transition: output RECEIVE ( search , 3, 7) in automaton sTreeNode (3) on 
nene . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (2 7), parent: 7, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

transition: output PARENT (7) in automaton sTreeNode (3) on 
nene . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (2 7), parent: 7, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND ( search , 3, 2) in automaton sTreeNode (3) on 
nene . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (2 7), parent: 7, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND(search, 14, 13) in automaton sTreeNode ( 14) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (10 13 15), parent: 10, reported: true, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runtime . adt . MapSort 

transition: output RECEIVE ( search , 14, 13) in automaton sTreeNode ( 14) on 
blackbird . csail .mit . edu 

Modified state variables: 

RM — > ioa . runtime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 13, 14) in automaton sTreeNode ( 13) on 
tui . csail .mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 15, 11) in automaton sTreeNode ( 15) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (11 14), parent: 11, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 
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transition: output PARENT (11) in automaton sTreeNode ( 15) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs : (11 14), parent: 11, reported: true, send: 
ioa . runtime . adt . Map Sort] 

transition: output SEND ( search , 15, 14) in automaton sTreeNode ( 15) on 
blackbird . csail .mit . edu 

Modified state variables: 

P — > [nbrs: (11 14), parent: 11, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 15, 14) in automaton sTreeNode ( 15) on 
blackbird . csail .mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 14, 15) in automaton sTreeNode ( 14) on 
blackbird . csail .mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 



Begin initialization 

Modified state variables: 

P — > [nbrs: (), parent: 87, reported: true, send: ioa . runt ime . adt . MapSort ] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i — » null 

End initialization 

transition: input initialize () in automaton sTreeNode (2) on 
condor . csail . mit . edu 

Modified state variables: 

P — > [nbrs: (1 3 6), parent: -1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i -> 2 

transition: output RECEIVE ( search , 2, 1) in automaton sTreeNode (2) on 
condor . csail . mit . edu 

Modified state variables: 

P — > [nbrs: (1 3 6), parent: 1, reported: false, send: 
ioa . runtime . adt . MapSort] 

RM — > ioa . runt ime . adt . MapSort 

transition: output PARENT (1) in automaton sTreeNode (2) on 
condor . csail . mit . edu 

Modified state variables: 

P — > [nbrs: (1 3 6), parent: 1, reported: true, send: 
ioa . runtime . adt . MapSort] 

transition: output SEND (search, 2, 3) in automaton sTreeNode (2) on 
condor . csail . mit . edu 
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Modified state variables: 

P — > [nbrs : (1 3 6), parent: 1, reported: true, send: 
ioa . runtime . adt . Map Sort] 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 2, 3) in automaton sTreeNode (2) on 
condor . csail . mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND ( search , 2, 6) in automaton sTreeNode (2) on 
condor . csail . mit . edu 

Modified state variables: 

P — > [nbrs: (1 3 6), parent: 1, reported: true, send: 
ioa . runtime . adt . MapSort] 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 6, 2) in automaton sTreeNode (6) on 
tui . csail .mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 2, 6) in automaton sTreeNode (2) on 
condor . csail . mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE ( search , 3, 2) in automaton sTreeNode (3) on 
nene . csail .mit . edu 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 



E.3 Asynchronous Broadcast Convergecast on 16 nodes 
Complete trace 

Begin initialization 

Modified state variables: 

P — > [val: 87, acked : (), nbrs: (), parent: 87, reported: true, send: 
ioa . runtime . adt . MapSort , 

temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i -> 

End initialization 

transition: input initialize () in automaton bcastNode 

Modified state variables: 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 1, 0) in automaton 
bcastNode 

P — > [val: -1, acked: (), nbrs: (0 5 8), parent: -1, reported: false, 
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send : ioa . runtime . adt . MapSort 
RM — > ioa . runtime . adt . MapSort 
SM — » ioa . runt ime . adt . MapSort 

i -> 4 



temp: [kind: beast, 



87]] 



Modified state variables: 

P — > [val: 99, acked : (), nbrs : (0 2 5), parent: 0, reported: false. 

send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 



99]), 1, 2) in automaton 



transition: output SEND (msg ( [kind : beast, 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (0 2 5), parent: 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, 
SM — > ioa . runt ime . adt . MapSort 



reported: false, 
w: 87]] 



transition: output SEND (msg ( [kind : beast, 
bcastNode 

Modified state variables: 
P — > [val: 99, acked: (), nbrs 
send : ioa . runtime . adt . MapSort , 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 



3] ) , 1, 5) in automaton 



(0 2 5), parent: 0, reported: false, 
temp: [kind: beast, w: 87]] 



: ( ) , parent 
temp : [kind : 



Begin initialization 

Modified state variables: 

P — > [val: 87, acked: (), nbrs 

send : ioa . runtime . adt . MapSort , 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i -> 

End initialization 

transition: input initialize () in automaton bcastNode 

Modified state variables: 

P — > [val: -1, acked: (), nbrs 

send : ioa . runtime . adt . MapSort , 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i -> 14 



87, reported: true 
beast, w: 87]] 



(10 13 15), parent: -1, reported: false, 
temp: [kind: beast, w: 87]] 



transition: output RECEIVE (msg ( [kind : beast, w: 991), 4, 0) in automaton 
bcastNode 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 5, 1) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (14 6 9), parent: 1, reported: false, 

send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 5, 4) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (14 6 9), parent: 1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runt ime . adt . MapSort 
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Modified state variables: 

P — > [val : 99, acked : (), nbrs : (0 5 8), parent: 0, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 4, 8) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (0 5 8), parent: 0, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 5, 9) in automaton 
bcastNode 

transition: output SEND (msg ( [kind : beast, w: 99]), 4, 5) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (0 5 8), parent: 0, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 4, 5) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (0 5 8), parent: 0, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 4, 5) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (), nbrs: (0 5 8), parent: 0, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runtime . adt . MapSort 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (14 6 9), parent: 1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 5, 6) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (14 6 9), parent: 1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 5, 4) in automaton 

bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (14 6 9), parent: 1, reported: false, 
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send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 5, 4) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked : (), nbrs : (14 6 9), parent: 1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 5, 4) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (4), nbrs: (14 6 9), parent: 1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 8, 4) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (12 4 9), parent: 4, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 8, 9) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (12 4 9), parent: 4, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — » ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 4, 5) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (5), nbrs: (0 5 8), parent: 0, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 2, 1) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (1 3 6), parent: 1, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 2, 3) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (1 3 6), parent: 1, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 8, 12) in automaton 
bcastNode 

Modified state variables: 



82 



P — > [val: 99, acked : (), nbrs : (12 4 9), parent: 4, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

Begin initialization 

Modified state variables: 

P — > [val: 87, acked: (), nbrs: (), parent: 87, reported: true, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i -> 

End initialization 

transition: input initialize () in automaton bcastNode 

transition: output SEND (msg ( [kind : beast, w: 99]), 2, 6) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (1 3 6), parent: 1, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

Modified state variables: 

P — > [val: -1, acked: (), nbrs: (10 2 5 7), parent: -1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 
i -> 6 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 6, 2) in automaton 

bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 2 5 7), parent: 2, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 6, 10) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 2 5 7), parent: 2, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runtime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 3, 2) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (2 7), parent: 2, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runtime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 3, 7) in automaton 
bcastNode 

transition: output SEND (msg ( [kind : beast, w: 99]), 6, 7) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 2 5 7), parent: 2, reported: false, 
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send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 6, 5) in automaton 
bcastNode 

Modified state variables: 

P — ► [val : 99, acked : (), nbrs : (10 2 5 7), parent: 2, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 6, 5) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 2 5 7), parent: 2, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runtime . adt . MapSort 

transition: output SEND (kind (ack ) , 6, 5) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 2 5 7), parent: 2, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runtime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 12, 8) in automaton 

bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (13 8), parent: 8, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 12, 13) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (13 8), parent: 8, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 5, 6) in automaton 
bcastNode 

Modified state variables: 

P — » [val: 99, acked: (4), nbrs: (14 6 9), parent: 1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 5, 6) in automaton bcastNode 

Modified state variables: 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (2 7), parent: 2, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runt ime . adt . MapSort 

P — > [val: 99, acked: (4), nbrs: (14 6 9), parent: 1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w : 87 ]] 
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RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 5, 6) in automaton bcastNode 
Modified state variables: 

P — > [val : 99, acked : (4 6), nbrs : (14 6 9), parent: 1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 6, 5) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (5), nbrs: (10 2 5 7), parent: 2, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 13, 12) in automaton 
bcastNode 

Modified state variables: 

P — * [val: 99, acked: (), nbrs: (12 14 9), parent: 12, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 13, 9) in automaton 

bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (12 14 9), parent: 12, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — » ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 13, 14) in automaton 

bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (12 14 9), parent: 12, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 7, 3) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (11 3 6), parent: 3, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 7, 11) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (11 3 6), parent: 3, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 7, 6) in automaton 
bcastNode 
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Modified state variables: 

P — > [val : 99, acked : (), nbrs : (11 3 6), parent: 3, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 7, 6) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (11 3 6), parent: 3, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — » ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 7, 6) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (), nbrs: (11 3 6), parent: 3, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 6, 7) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (5), nbrs: (10 2 5 7), parent: 2, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 6, 7) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (5), nbrs: (10 2 5 7), parent: 2, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 6, 7) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (5 7), nbrs: (10 2 5 7), parent: 2, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 7, 6) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (6), nbrs: (11 3 6), parent: 3, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 11, 7) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 15 7), parent: 7, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 
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transition: output SEND (msg ( [kind : beast, w: 99]), 11, 10) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked : (), nbrs : (10 15 7), parent: 7, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 14, 13) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 13 15), parent: 13, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runtime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 14, 10) in automaton 
bcastNode 

Modified state variables: 

P — » [val: 99, acked: (), nbrs: (10 13 15), parent: 13, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 11, 15) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 15 7), parent: 7, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 14, 15) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 13 15), parent: 13, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — » ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 11, 10) in automaton 
bcastNode 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 10, 6) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (11 14 6 9), parent: 6, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 10, 11) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (11 14 6 9), parent: 6, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 10, 11) in automaton 
bcastNode 

Modified state variables: 
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P — > [val : 99, acked : (), nbrs : (11 14 6 9), parent: 6, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 10, 14) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (11 14 6 9), parent: 6, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 10, 14) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (11 14 6 9), parent: 6, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runtime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 10, 9) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (11 14 6 9), parent: 6, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runtime . adt . MapSort 



Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 15 7), parent: 7, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 11, 10) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 15 7), parent: 7, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 14, 10) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 13 15), parent: 13, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 14, 10) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 13 15), parent: 13, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 



transition: output SEND (kind (ack ) , 10, 11) in automaton bcastNode 
Modified state variables: 

P — > [val : 99, acked : (), nbrs : (11 14 6 9), parent: 6, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 11, 10) in automaton bcastNode 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 15, 11) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (11 14), parent: 11, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runtime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 15, 14) in automaton 
bcastNode 

Modified state variables: 

P — ► [val: 99, acked: (), nbrs: (11 14), parent: 11, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 15, 14) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (11 14), parent: 11, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 15, 14) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (), nbrs: (11 14), parent: 11, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runtime . adt . MapSort 

Modified state variables: 

P — > [val: 99, acked: (10), nbrs: (10 15 7), parent: 7, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 10, 11) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (11), nbrs: (11 14 6 9), parent: 6, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 10, 14) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (11), nbrs: (11 14 6 9), parent: 6, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
SM — > ioa . runt ime . adt . MapSort 
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transition: output RECEIVE (kind (ack) , 10, 14) in automaton bcastNode 
Modified state variables: 

P -> [val: 99, acked : (11 14), nbrs : (11 14 6 
false, send: ioa . runt ime . adt . MapSort , temp: [kind: 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 



9), parent: 6, reported: 
beast, w: 87]] 



transition: output RECEIVE (kind (ack) , 14, 10) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (10), nbrs: (10 13 15), parent: 13, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 14, 15) in automaton 
bcastNode 

Modified state variables: 

P — ► [val: 99, acked: (10), nbrs: (10 13 15), parent: 13, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 14, 15) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (10), nbrs: (10 13 15), parent: 13, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 14, 15) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (10 15), nbrs: (10 13 15), parent: 13, reported: 
false, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 



Begin initialization 

Modified state variables: 

P — > [val: 87, acked: (), nbrs: (), parent: 87. 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runtime . adt . MapSort 

i -> 

End initialization 

transition: input initialize () in automaton bcastNode 

Modified state variables: 

P — > [val: -1, acked: (), nbrs: (10 13 5 8), parent: -1 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runtime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

i -> 9 



reported: true, send: 



reported: false. 



transition: internal report (14) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (10 15), nbrs: (10 13 15), parent: 13, reported: 
true, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
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transition: output SEND (kind (ack ) , 14, 13) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked : (10 15), nbrs : (10 13 15), parent: 13, reported: 
true, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 13, 14) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (14), nbrs: (12 14 9), parent: 12, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — » ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 15, 14) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (14), nbrs: (11 14), parent: 11, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: internal report (15) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (14), nbrs: (11 14), parent: 11, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 15, 11) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (14), nbrs: (11 14), parent: 11, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
SM — » ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 11, 15) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (10 15), nbrs: (10 15 7), parent: 7, reported: 
false, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 

transition: internal report (11) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (10 15), nbrs: (10 15 7), parent: 7, reported: true 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 11, 7) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (10 15), nbrs: (10 15 7), parent: 7, reported: true 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 7, 11) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (11 6), nbrs: (11 3 6), parent: 3, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 

transition: internal report (7) in automaton bcastNode 
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Modified state variables: 

P — > [val : 99, acked : (11 6), nbrs : (11 3 6), parent: 3, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 7, 3) in automaton bcastNode 
transition: output RECEIVE (kind (ack) , 3, 7) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (7), nbrs: (2 7), parent: 2, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: internal report (3) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (7), nbrs: (2 7), parent: 2, reported: true, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 3, 2) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (7), nbrs: (2 7), parent: 2, reported: true, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
SM — > ioa . runt ime . adt . MapSort 

Modified state variables: 

P — > [val: 99, acked: (11 6), nbrs: (11 3 6), parent: 3, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 2, 3) in automaton bcastNode 

Modified state variables: 

P — > [val: 99, acked: (3), nbrs: (1 3 6), parent: 1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runtime . adt . MapSort transition: output RECEIVE (msg ( [kind : 
beast, w: 991), 9, 5) in automaton bcastNode 



Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 13 5 8), parent: 5, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 9, 13) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 13 5 8), parent: 5, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runtime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 9, 8) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 13 5 8), parent: 5, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 
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SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 13, 9) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked : (14), nbrs : (12 14 9), parent: 12, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — * ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 13, 9) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (14), nbrs: (12 14 9), parent: 12, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 9, 8) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 13 5 8), parent: 5, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (msg ( [kind : beast, w: 99]), 9, 10) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 13 5 8), parent: 5, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 10, 9) in automaton 

bcastNode 

Modified state variables: 

P — > [val: 99, acked: (11 14), nbrs: (11 14 6 9), parent: 6, reported: 
false, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runtime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 9, 10) in automaton 
bcastNode 



Modified state variables: 

transition: output SEND (kind (ack ) , 10, 9) in automaton bcastNode 

P — > [val: 99, acked: (), nbrs: (10 13 5 8), parent: 5, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

Modified state variables: 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runtime . adt . MapSort P — > [val: 99, acked: (11 14), nbrs: 
(11 14 6 9), parent: 6, reported: false, send: ioa . runt ime . adt . MapSort , temp: 
[kind: beast, w: 87]] 

RM — ► ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 99]), 9, 13) in automaton 
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bcastNode 

Modified state variables: 

P — > [val: 99, acked : (), nbrs : (10 13 5 8), parent: 5, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 9, 8) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 13 5 8), parent: 5, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 9, 10) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (), nbrs: (10 13 5 8), parent: 5, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 10, 9) in automaton bcastNode 

Modified state variables: 

transition: output RECEIVE (kind (ack) , 9, 10) in automaton bcastNode 

Modified state variables: 

P — > [val: 99, acked: (10), nbrs: (10 13 5 8), parent: 5, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

P — > [val: 99, acked: (11 14 9), nbrs: (11 14 6 9), parent: 6, reported: 
false , send : ioa . runtime . adt . MapSort , 

temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort RM — > ioa . runt ime . adt . MapSort 

SM — » ioa . runt ime . adt . MapSort SM — > ioa . runt ime . adt . MapSort 



transition: internal report (10) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (11 14 9), nbrs: (11 14 6 9), parent: 6, reported: 
true, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 10, 6) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (11 14 9), nbrs: (11 14 6 9), parent: 6, reported: 
true, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
SM — > ioa . runtime . adt . MapSort 
transition: output SEND (kind (ack ) , 9, 13) in automaton bcastNode 

Modified state variables: 

P — > [val: 99, acked: (10), nbrs: (10 13 5 8), parent: 5, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 9, 13) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (10 13), nbrs: (10 13 5 8), parent: 5, reported: 
false, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
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RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 13, 9) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked : (14 9), nbrs : (12 14 9), parent: 12, reported: 
false, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: internal report (13) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (14 9), nbrs: (12 14 9), parent: 12, reported: true., 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 13, 12) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (14 9), nbrs: (12 14 9), parent: 12, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 12, 13) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (13), nbrs: (13 8), parent: 8, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: internal report (12) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (13), nbrs: (13 8), parent: 8, reported: true, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 12, 8) in automaton bcastNode 
Modified state variables: 

P — ► [val: 99, acked: (13), nbrs: (13 8), parent: 8, reported: true, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (msg ( [kind : beast, w: 991), 8, 9) in automaton 
bcastNode 

Modified state variables: 

P — > [val: 99, acked: (), nbrs: (12 4 9), parent: 4, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

SM — > ioa . runt ime . adt . MapSort 

transition: output SEND (kind (ack ) , 8, 9) in automaton bcastNode 
Modified state variables: 

transition: output RECEIVE (kind (ack) , 6, 10) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (10 5 7), nbrs: (10 2 5 7), parent: 2, reported: 
false, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
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transition : internal report (6) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked : (10 5 7), nbrs : (10 2 5 7), parent: 2, reported: 
true, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 6, 2) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (10 5 7), nbrs: (10 2 5 7), parent: 2, reported: 
true, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 2, 6) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (3 6), nbrs: (1 3 6), parent: 1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 

transition: internal report (2) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (3 6), nbrs: (1 3 6), parent: 1, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 2, 1) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (3 6), nbrs: (1 3 6), parent: 1, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 1, 2) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (2), nbrs: (0 2 5), parent: 0, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

P — > [val: 99, acked: (), nbrs: (12 4 9), parent: 4, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 9, 8) in automaton bcastNode 
Modified state variables: 

transition: output RECEIVE (kind (ack) , 8, 9) in automaton bcastNode 
P — > [val: 99, acked: (10 13 8), nbrs: (10 13 5 8), parent: 5, reported: 
false, send: ioa . runt ime . adt . MapSort , 
temp: [kind: beast, w: 87]] 

RM — > ioa . runt ime . adt . MapSort 

Modified state variables: 

transition: internal report (9) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (9), nbrs: (12 4 9), parent: 4, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
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SM — > ioa . runt ime . adt . MapSort 

P — > [val: 99, acked : (10 13 8), nbrs : (10 13 5 8), parent: 5, reported: 
true, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 9, 5) in automaton bcastNode 

Modified state variables: 

P — > [val: 99, acked: (10 13 8), nbrs: (10 13 5 8), parent: 5, reported: 
true, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 8, 12) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (12 9), nbrs: (12 4 9), parent: 4, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 

transition: internal report (8) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (12 9), nbrs: (12 4 9), parent: 4, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 8, 4) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (12 9), nbrs: (12 4 9), parent: 4, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 5, 9) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (4 6 9), nbrs: (14 6 9), parent: 1, reported: 
false, send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 

transition: internal report (5) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (4 6 9), nbrs: (14 6 9), parent: 1, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 5, 1) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (4 6 9), nbrs: (14 6 9), parent: 1, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 1, 5) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (2 5), nbrs: (0 2 5), parent: 0, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runt ime . adt . MapSort 

transition: internal report (1) in automaton bcastNode 
Modified state variables: 
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P — > [val: 99, acked : (2 5), nbrs : (0 2 5), parent: 0, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 1, 0) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (2 5), nbrs: (0 2 5), parent: 0, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
SM — * ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 4, 8) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (5 8), nbrs: (0 5 8), parent: 0, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 

transition: internal report (4) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (5 8), nbrs: (0 5 8), parent: 0, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 

transition: output SEND (kind (ack ) , 4, 0) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (5 8), nbrs: (0 5 8), parent: 0, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 87]] 
RM — > ioa . runtime . adt . MapSort 
SM — > ioa . runtime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 0, 1) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (1), nbrs: (1 4), parent: -1, reported: false, send: 
ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 99]] 
RM — > ioa . runt ime . adt . MapSort 
SM — > ioa . runt ime . adt . MapSort 

transition: output RECEIVE (kind (ack) , 0, 4) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (1 4), nbrs: (1 4), parent: -1, reported: false, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 99]] 
RM — > ioa . runt ime . adt . MapSort 

transition: internal report (0) in automaton bcastNode 
Modified state variables: 

P — > [val: 99, acked: (1 4), nbrs: (1 4), parent: -1, reported: true, 
send: ioa . runt ime . adt . MapSort , temp: [kind: beast, w: 99]] 



E.4 Spanning Tree to Leader Election on 16 nodes 
Complete trace 

Trace of sTreeLeader 

Number of machines: 5 
Number of nodes : 16 
Duration : 9 sec 
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Number of messages exchanged: 16 



on loon.csail.mit.edu at 9:29:05:372 



(), receivedElect: (), sentElect: (), status: idle, send: Map {}] 



Initialization starts (15) 
Modified state variables: 
P -> 
RM -i 

SM -i 

J -» 
k -> 
rank 



[nbrs : 
Map-C} 
Map{} 
87 
87 

-> null 
tempNbrs — > () 
tempNbrs2 — > () 
Initialization ends 
transition: output SEND(elect, 
on loon.csail.mit.edu at 9:29:05:3 
Modified state variables: 
P 



15, 11) in automaton sTreeLeader (15) 



[nbrs : (11) , receivedElect : () , sentElect : () , status 
RM — > Map-[[11 -> [status: idle, toRecv: {}, ready: false]] 
SM — > Map-C [11 -> [status: idle, toSend : {elect}, sent: {}, 

j -» 11 
k -> 11 
rank — > 15 



idle, send: Map{[ll -> {}] }] 



handles: {}] ] } 



tempNbrs 
tempNbr s2 



() 



() 



Initialization starts (0) on loon.csail.mit.edu at 9:29:05:847 

Modified state variables: 

P — > [nbrs : () , receivedElect : () , sentElect : () , status : idle , send : Map-C}] 

RM -h 
SM -i 

j -» 
k -> 
rank 



[nbrs : 
Map-[} 
Map-C} 

87 

87 

-> null 
tempNbrs — ► () 
tempNbrs2 — > () 
Initialization ends 
transition: output SEND(elect 
on loon.csail.mit.edu at 9:29:05:860 
Modified state variables: 
P — > [nbrs : (1) , receivedElect : 
RM — > Map-C [1 -> [status: idle 
SM — ► Map{[l -> [status: idle 

j -► 1 
k -> 1 
rank — > 



0, 1) in automaton sTreeLeader (0) 



() , sentElect : () , status 
toRecv: {}, ready: false]] 
toSend: {elect}, sent: {}, 



idle, send: Map{[l -> {}] }] 



handles: {}]] } 



tempNbrs 
t empNbr s2 



() 



() 



Initialization starts (14) on tui.csail.mit.edu at 9:29:05:977 

Initialization starts (3) on parrot.csail.mit.edu at 9:29:05:978 

Modified state variables: 

Modified state variables: 

P — > [nbrs : () , receivedElect : () , sentElect : () , status : idle , send : Map{}] 

RM -> Map{} 

SM -> Map{} 
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[nbrs : () , receivedElect : () , sentElect : () , status : idle , send : MapO] 



k -> 87 




RM -> MapO 




tempNbrs — ► 


() 


tempNbrs2 — » 


() 


SM -> MapO 




j -> 87 




k -> 87 





rank — > null 



Initialization ends 



rank — > null 

tempNbrs — > () 

tempNbrs2 — > () 

Initialization ends 

Initialization starts (13) on parrot.csail.mit.edu at 9:29:06:210 

Modified state variables: 

P — > [nbrs : () , receivedElect : () , sentElect : () , status : idle , send : MapO] 

RM -> MapO 

Initialization starts (8) on parrot.csail.mit.edu at 9:29:06:475 

Initialization starts (11) on drake.csail.mit.edu at 9:29:06:477 

Modified state variables: 

t : () , sentElect : () , status : idle , send : MapO] 



p -> 


[nbrs: (), receivedE 


RM - 


+ MapO 


SM - 


+ MapO 


j -> 


87 


k -> 


87 


rank 


-> null 


temp 


Nbrs -> () 


temp 


Nbrs2 -> () 


Init 


ialization ends 


Modi 


fied state variables: 


P -> 


[nbrs: (), receivedE 


RM - 


+ MapO 


SM - 


+ MapO 


j -> 


87 


k -> 


87 


rank 


-> null 



(), sentElect: (), status: idle, send: Map{}] 



tempNbrs — > () 
tempNbrs2 —* () 
Initialization ends 
SM -> Map{> 
j -► 87 
k -» 87 
rank — » null 
tempNbrs — » () 
tempNbrs2 — » () 
Initialization ends 

transition: output SEND(elect, 13, 14) in automaton sTreeLeader (13) 
on parrot.csail.mit.edu at 9:29:06:796 
Modified state variables: 

P — > [nbrs: (14), receivedElect: (), sentElect: (), status: idle, send: Map{[14 -> {}] }] 
RM — » Map{[14 -> [status: idle, toRecv: {}, ready: false]] > 

SM — » Map{[14 -> [status: idle, toSend : {elect}, sent: {}, handles: {}] ] } 
j -» 14 
k -» 14 
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rank — > 13 
tempNbrs — > () 
tempNbrs2 — > () 

Initialization starts (1) on drake.csail.mit.edu at 9:29:07:059 
Initialization starts (2) on condor.csail.mit.edu at 9:29:07:062 
Modified state variables: 

Initialization starts (4) on tui.csail.mit.edu at 9:29:07:150 
Modified state variables: 

P — > [nbrs : () , receivedElect : () , sentElect : () , status : idle , send : Map{}] 
RM -> Map-D 
SM -> Map{} 
j -► 87 
k -> 87 
rank — > null 
tempNbrs — ► () 
tempNbrs2 — > () 
Initialization ends 

transition: output RECEIVE ( elect , 11, 15) in automaton sTreeLeader (11) 
on drake.csail.mit.edu at 9:29:07:180 
Modified state variables: 

P — » [nbrs: (15 7), receivedElect: (15), sentElect: (7), status: idle, send: Map{[15 -> {}] 
RM — > Map{[15 -> [status: idle, toRecv: {}, ready: false]] [7 -> [status: idle, toRecv: {}, 
SM — > Map{[15 -> [status: idle, toSend : {}, sent: {} , handles: {}]] [7 -> [status: idle, tc 
j -» 7 
k -> 15 
rank — > 11 
tempNbrs — ► () 
tempNbrs2 — > () 

Initialization starts (6) on drake.csail.mit.edu at 9:29:07:304 
Modified state variables: 

transition: output SEND(elect, 4, 5) in automaton sTreeLeader (4) 
on tui.csail.mit.edu at 9:29:07:401 
Modified state variables: 

: (), sentElect: (), status: idle, send: Map-[[5 -> {}] }] 
toRecv: {}, ready: false]] } 
toSend: {elect}, sent: {}, handles: {}]] } 



p -> 


[nbrs : 


(5), 


receivedElect 


RM - 


+ Map{[5 


-> 


[ status : idle , 


SM - 


+ Map {[5 


-> 


[ status : idle , 


j -► 


5 






k -> 


5 






rank 


-> 4 






temp 


Nbrs — > 


() 




temp 


Nbrs2 -> 


() 





transition: output SEND (elect, 11, 7) in automaton sTreeLeader (11) 
on drake.csail.mit.edu at 9:29:07:485 
Modified state variables: 

P — » Tuple, modified fields: {[send -> Map, modified entries: {[7 -> Sequence, elements adc 
SM — > Map, modified entries: {[7 -> Tuple, modified fields: {[toSend -> Sequence, elements 
k -> 7 

Initialization starts (7) on condor.csail.mit.edu at 9:29:07:260 

Modified state variables: 

P — > [nbrs : () , receivedElect : () , sentElect : () , status : idle , send : Map{}] 

RM -» Map{> 

SM -» Map{> 
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j -» 87 
k -> 87 
rank — > null 
tempNbrs — ► () 
tempNbrs2 — > () 
Initialization ends 
Initialization 



starts (12) on condor.csail.mit.edu at 9:2 
Modified state variables: 



P — > [nbrs : () , receivedElect : () , sentElect : () 

RM -> Map{> 

SM -> Map{} 

j -» 87 

k -> 87 

rank — > null 

tempNbrs — ► () 

tempNbrs2 — > () 

Initialization ends 

P — > [nbrs : 

RM -> Map{> 

SM -> Map{> 

j -> 87 

k -> 87 

rank — > null 

tempNbrs — > () 

tempNbrs2 — > () 

Initialization ends 

Initialization 



9:07:712 
status: idle, send: Map{}] 



(), receivedElect: (), sentElect: (), status: idle, send: Map {}] 



starts (10) on loon . csail . mit . 
Modified state variables: 



P -> 
RM - 
SM - 

j -> 
k -» 
rank 



[nbrs : 
. Map-O 
■ Map-O 

87 

87 

-^ null 



(), receivedElect 



edu at 9:29:07:954 
(), sentElect: (), status: idle, send: Map{}] 



() 



tempNbrs 
tempNbrs2 — > () 
Initialization ends 
P — » [nbrs : () , 
RM -» Map{> 
SM -» Map{> 
j -» 87 
k ^ 87 
rank — » null 
tempNbrs — » () 
tempNbrs2 — » () 
Initialization ends 
Initialization 
Modified state 
P -» 
RM - 
SM - 

j "^ 
k -> 
rank 



nds 

receivedElect : () , sentElect : () , status : idle , send : 



Map{}] 



[nbrs : 
Map-O 
Map-O 

87 

87 

-> null 



() 



starts (9) on tui.csail.mit.edu at 9:29:08:081 
variables : 
receivedElect : () , sentElect : () , status : idle 



send : Map{}] 



tempNbrs 
tempNbr s2 



() 
• () 
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(), receivedElect: (), sentElect: (), status: idle, send: Map{}] 



Initialization ends 
Modified state variables: 

P -» 
RM -i 
SM — 

J -> 
k -> 
rank 



[nbrs : 
Map{} 
Map{} 
87 
87 

-> null 
tempNbrs — > () 
tempNbrs2 — > () 
Initialization ends 
transition: output SEND(elect, 
on condor.csail.mit.edu at 9:29:08:395 
Modified state variables: 
P — > [nbrs: (8), receivedElect 
RM — > Map{[8 -> [status: idle, 
Map {[8 -> 



12, 8) in automaton sTreeLeader (12) 



SM -i 

J -> 
k -> 
rank 



[ status : idle 



(), sentElect: (), status 
toRecv: {}, ready: false]] 
toSend: {elect}, sent: {}, 



idle, send: Map-[[8 -> {}] }] 



handles: {}]] > 



12 



tempNbrs 
tempNbrs2 



() 



() 



transition: output RECEIVE ( elect , 1, 0) in automaton sTreeLeader (1) 
on drake.csail.mit.edu at 9:29:08:496 

transition: output RECEIVE ( elect , 14, 13) in automaton sTreeLeader (14) 
on tui.csail.mit.edu at 9:29:08:528 

Modified state variables: 

P — > [nbrs: (10 13), receivedElect: (13), sentElect: (10), status: idle, send: Map{[10 -> { 

RM — > Map{[10 -> [status: idle, toRecv: {}, ready: false]] [13 -> [status: idle, toRecv: {] 

SM — > Map{[10 -> [status: idle, toSend : {}, sent: {}, handles: {}]] [13 -> [status: idle, 1 

j -» 10 

k -> 13 

rank — > 14 



tempNbrs 
tempNbr s2 



() 



() 



transition: output SEND(elect, 14, 10) in automaton sTreeLeader (14) 
on tui.csail.mit.edu at 9:29:08:536 
Modified state variables: 

P — > Tuple, modified fields: {[send -> Map, modified entries: {[10 -> 
SM — > Map, modified entries: {[10 -> Tuple, modified fields: {[toSend 
k -> 10 
tempNbrs2 — > (13) 



Sequence, elements ac 
-> Sequence, elements 



Modified state variables: 
P — > [nbrs : (0 5), receivedElect : (0) , 
RM — > Map{[0 -> [status: idle, toRecv: 
SM — > Map{[0 -> [status: idle, toSend: 

j -» 
k -> 
rank — > 1 



tempNbrs 
tempNbr s2 



() 



(5) 



sentElect: (5), status: idle, send: Map{[0 -> {}] [E 
{}, ready: false]] [5 -> [status: idle, toRecv: {}, 
{}, sent: {}, handles: {}]] [5 -> [status: idle, toS 



transition: output SEND (elect, 1, 5) in automaton sTreeLeader (1) 
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on drake.csail.mit.edu at 9:29:08:965 
Modified state variables: 

P — > Tuple, modified fields: {[send -> Map, modified entries: {[5 -> Sequence, elements adc 
SM — > Map, modified entries: {[5 -> Tuple, modified fields: -{[toSend -> Sequence, elements 

k -> 5 
tempNbrs2 — > (0) 



transi 

on condor . 
Modif i 
P -> [ 

RM -» 
SM -> 
J - 3 
k -> 1 
rank - 
tempNb 
tempNb 



tion : output RECEIVE ( elect , 7, 11) in automaton sTreeLeader (7) 

csail.mit.edu at 9:29:09:760 

ed state variables : 

nbrs: (11 3), receivedElect: (11), sentElect: (3), status: idle, send: Map {[11 -> 

Map{[ll -> [status: idle, toRecv: {}, ready: false]] [3 -> [status: idle, toRecv: 

Map{[ll -> [status: idle, toSend : {}, sent: {>, handles: {}]] [3 -> [status: idle. 



1 

-» 7 
r s - 
rs2 



() 



{}] 
O, 

tc 



() 



transition: output SEND(elect, 7, 3) in automaton sTreeLeader (7) 
on condor.csail.mit.edu at 9:29:09:780 
Modified state variables: 

P — > Tuple, modified fields: {[send -> Map, modified entries: {[3 -> Sequence, elements adc 
SM — > Map, modified entries: {[3 -> Tuple, modified fields: {[toSend -> Sequence, elements 
k -> 3 



transition: output RECEIVE ( elect 
on loon.csail.mit.edu at 9:29:10:113 
Modified state variables: 



10, 14) in automaton sTreeLeader (10) 



P — > [nbrs: (14 6), receivedElect: (14) 
RM — > Map{[14 -> [status: idle, toRecv: 
SM — > Map{[14 -> [status: idle, toSend: 

j -» 14 
k -> 14 
rank — > 10 



tempNbrs 
tempNbr s2 



() 



(6) 



sentElect: (6), status: idle, send: Map{[14 -> 
{}, ready: false]] [6 -> [status: idle, toRecv: 
{}, sent: {}, handles: {}]] [6 -> [status: idle.. 



{}] 
O, 

tc 



transition: output SEND(elect, 10, 6) in automaton sTreeLeader (10) 
on loon.csail.mit.edu at 9:29:10:119 
Modified state variables: 
P — > Tuple, modified fields: {[send -> Map, modified entries: {[6 -> Sequence, elements adc 



SM -> 
k -> 6 
tempNbr s2 



Map, modified entries: {[6 -> Tuple, modified fields: {[toSend -> Sequence, elements 
() 



transition: output RECEIVE ( elect , 6, 10) in automaton sTreeLeader (6) 
on drake.csail.mit.edu at 9:29:10:522 
Modified state variables: 

P — > [nbrs: (10 2 5), receivedElect: (10), sentElect: (), status: idle, send: Map{[10 -> {] 
RM — > Map{[10 -> [status: idle, toRecv: {}, ready: false]] [2 -> [status: idle, toRecv: {>, 
SM — > Map{[10 -> [status: idle, toSend: {}, sent: {}, handles: {}]] [2 -> [status: idle, tc 
j -» 10 
k -> 10 
rank — > 6 



tempNbrs 



() 
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tempNbr s2 



() 



transi 

on parrot . 
Modif i 

P -> [ 
RM -> 
SM -> 

i -» i 

k -> 1 
rank - 
tempNb 
tempNb 



tion : output RECEIVE (elect , 8, 12) in automaton sTreeLeader (8) 

csail.mit.edu at 9:29:11:670 

ed state variables : 

nbrs : (12 9), receivedElect : (12), sentElect : (9), status: idle, send: Map{[12 -> {}] 

Map{[12 -> [status: idle, toRecv: {}, ready: false]] [9 -> [status: idle, toRecv: {}, 

Map{[12 -> [status: idle, toSend : {}, sent: {}, handles: {}]] [9 -> [status: idle, tc 

2 

2 



r s 
rs2 



() 



() 



transition: output SEND(elect, 8, 9) in automaton sTreeLeader (8) 
parrot.csail.mit.edu at 9:29:11:684 
Modified state variables: 
P — > Tuple, modified fields: {[send 
SM — > Map, modified entries: {[9 -> 
k -> 9 
tempNbrs2 — > (12) 



-> Map, modified entries: {[9 -> Sequence, elements adc 
Tuple, modified fields: -[[toSend -> Sequence, elements 



transi 

on parrot . 
Modif i 
P - [ 

RM -> 
SM -> 
j -» 7 
k -> 7 
rank - 
tempNb 
tempNb 



tion: output RECEIVE ( elect , 3, 7) in automaton sTreeLeader (3) 

csail.mit.edu at 9:29:12:228 

ed state variables : 

nbrs : (2 7), receivedElect: (7), sentElect: (2), status: idle, send: Map {[2 -> {elect 

Map{[2 -> [status: idle, toRecv: {}, ready: false]] [7 -> [status: idle, toRecv: {}, 

Map{[2 -> [status: idle, toSend: {} , sent: {}, handles: {}] ] [7 -> [status: idle, toS 



■* 3 
r s - 
rs2 



() 



() 



transition: output SEND(elect, 3, 2) in automaton sTreeLeader (3) 
on parrot.csail.mit.edu at 9:29:12:262 

transition: output RECEIVE (elect , 9, 8) in automaton sTreeLeader (9) 
on tui.csail.mit.edu at 9:29:12:573 

Modified state variables: 

P — > [nbrs : (5 8), receivedElect : (8) , 

RM — > Map{[5 -> [status: idle, toRecv: 

SM — > Map{[5 -> [status: idle, toSend: 

j -» 5 

k -> 8 

rank — > 9 

tempNbr s 

tempNbr s2 



sentElect: (5), status: idle, send: Map{[5 -> {elect 
{}, ready: false]] [8 -> [status: idle, toRecv: {}, 
{}, sent: {>, handles: {}] ] [8 -> [status: idle, toS 



() 



() 



Modified state variables: 

P — > Tuple, modified fields: {[send -> Map, modified entries: {[2 -> Sequence, elements adc 

SM — > Map, modified entries: {[2 -> Tuple, modified fields: {[toSend -> Sequence, elements 

k -> 2 
tempNbrs2 — > (7) 

transition: output SEND(elect, 9, 5) in automaton sTreeLeader (9) 
on tui.csail.mit.edu at 9:29:12:730 
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Modified state variables: 

P — > Tuple, modified fields: {[send -> Map, modified entries: {[5 -> Sequence, elements adc 

SM — > Map, modified entries: {[5 -> Tuple, modified fields: -[[toSend -> Sequence, elements 

k -> 5 
tempNbrs2 — > (8) 



Initialization starts (5) on loon.csail.mit.edu at 9:29:12:852 
Modified state variables: 



P -> 
RM -i 
SM — 

J -> 
k -> 
rank 



() , receivedElect : () , sentElect : () , status : idle , send : Map{}] 



[nbrs : 
Map-O 
Map{} 
87 
87 

-> null 
tempNbrs — ► () 
tempNbrs2 — > () 
Initialization ends 
transition: output RECEIVE ( elect , 
on condor.csail.mit.edu at 9:29:14:748 
Modified state variables: 



2, 3) in automaton sTreeLeader (2) 



P 

RM 
SM 

j -> 3 
k -> 3 
rank — > 2 
tempNbrs 
tempNbr s2 



[nbrs : (3 6), receivedElect : (3) . 
Map{[3 -> [status: idle, toRecv 
Map{[3 -> [status: idle, toSend 



sentElect: (6), status: idle, send: Map{[3 -> {}] [6 
{}, ready: false]] [6 -> [status: idle, toRecv: {}, 
O, sent: {}, handles: {}]] [6 -> [status: idle, toS 



() 



() 



transition: output SEND(elect, 2, 6) in automaton sTreeLeader (2) 
on condor.csail.mit.edu at 9:29:14:759 
Modified state variables: 

P — > Tuple, modified fields: {[send -> Map, modified entries: {[6 -> Sequence, elements adc 
SM — > Map, modified entries: {[6 -> Tuple, modified fields: -[[toSend -> Sequence, elements 
k -> 6 

transition: output RECEIVE ( elect , 6, 2) in automaton sTreeLeader (6) 
on drake.csail.mit.edu at 9:29:15:104 
Modified state variables: 
P — > Tuple, modified fields: {[receivedElect -> (10 2)] [sentElect -> (5)] [send -> Map, mc 

k -> 2 

tempNbrs2 — > (10) 



transition: output SEND(elect, 6, 5) 
on drake.csail.mit.edu at 9:29:15:112 

transition: output RECEIVE ( elect , 5, 
on loon.csail.mit.edu at 9:29:15:145 

Modified state variables: 



P 

RM 

SM 

j -> 6 

k -> 1 

rank — > 5 

tempNbrs 

tempNbr s2 



in automaton sTreeLeader (6) 

1) in automaton sTreeLeader (5) 



[nbrs : (14 6 9), receivedElect : 
Map{[l -> [status: idle, toRecv 
Map{[l -> [status: idle, toSend 



() 



(6 9) 



(1), sentElect: (), status: idle, send: Map{[l -> {}] 
{}, ready: false]] [4 -> [status: idle, toRecv: {}, 
{>, sent: {>, handles: {>]] [4 -> [status: idle, toS 
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transition: output RECEIVE ( elect , 5, 4) in automaton sTreeLeader (5) 
on loon.csail.mit.edu at 9:29:15:492 
Modified state variables: 
P — > Tuple, modified fields: { [receivedElect -> (1 4)] } 

k -> 4 
tempNbrs2 — > () 

Modified state variables: 

P — > Tuple, modified fields: {[send -> Map, modified entries: {[5 -> Sequence, elements adc 

SM — > Map, modified entries: {[5 -> Tuple, modified fields: -{[toSend -> Sequence, elements 

k -> 5 

tempNbrs2 — > (10 2) 

transition: output RECEIVE ( elect , 5, 9) in automaton sTreeLeader (5) 
on loon.csail.mit.edu at 9:29:15:836 
Modified state variables: 

P — > Tuple, modified fields: {[receivedElect -> (1 4 9)] [sentElect -> (6)] [send -> Map, r 
k -> 9 

transition: output SEND(elect, 5, 6) in automaton sTreeLeader (5) 
on loon.csail.mit.edu at 9:29:16:182 
Modified state variables: 

P — > Tuple, modified fields: {[send -> Map, modified entries: {[6 -> Sequence, elements adc 
SM — > Map, modified entries: {[6 -> Tuple, modified fields: {[toSend -> Sequence, elements 
k -> 6 
tempNbrs2 — > (14 9) 

transition: output RECEIVE ( elect , 5, 6) in automaton sTreeLeader (5) 
on loon.csail.mit.edu at 9:29:16:494 
Modified state variables: 

P — > Tuple, modified fields: {[receivedElect -> (1 4 6 9)] } 
SM — > Map, modified entries: {[6 -> Tuple, modified fields: {[toSend -> Sequence, elements 

transition: output RECEIVE ( elect , 6, 5) in automaton sTreeLeader (6) 
on drake.csail.mit.edu at 9:29:16:554 
Modified state variables: 

P — > Tuple, modified fields: {[receivedElect -> (10 2 5)] [status -> elected] } 
SM — > Map, modified entries: {[5 -> Tuple, modified fields: {[toSend -> Sequence, elements 

transition: output leader () in automaton sTreeLeader (6) 
on drake.csail.mit.edu at 9:29:16:559 
Modified state variables: 
P — > Tuple, modified fields: {[status -> announced] } 
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